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Ibis document is one of a series of reports ^Mhich 
describe the experiences of six state agencies that are developln| 
systems to afreet the aeyelopment and implementation of policy i>-^the 
federal/state prograu for vocational rehabilitatioQ. ihe state 
pcgiams are developing model program OTaluation/nanagemeat 
inforaiation, support units. This second i,*suance of observations 
contains an outline supported by flow charts and summaries of 
Delajrare's case review process, Oregon's deployment of staff in 
tracking issues is described and illustrated with an example of the 
teGhnique at work. An interim report from Michigan examines' the 
prograo eval-uator's role as consultant through two case histories 
Virginia's approach to evaluation , uses team techingues employing both 
evaluation staff ^and program personnel. Pennsylvania's contribution 
aesciibes the structure of its case review process in terms of 
r|gional and district roles- west Virginia describes an ongbing study 

1. °* ""^3 benefit-cost analysis in vocational 

rehabilitation. Mississippi suggests that development of a management 
information system might well precede attempts at evaluatisn. Tha 
report concludes with a review, contributed by wimt Virginia, of a 
aew handbook m human services evaluation, and a bibliography- 
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PREFACE 



The primary purpose of this series of reoort^ le ^« 
experiences of sli State A^Bnri Z reports is to document the 

Agencies to dev^loV^od 1 ^ olr^^ evalJJtL'n/L'T' 

support units The eonfr^J,. ..f^ evaluation/management Information 

merit of pr^p;saS sZJttad InT °° "^"^^ °^ technical 

the States' rLelilnrthe"„1tJacLT«: " ' "^"""^ ^"^"^""^^ f-^' 

Delaware (a ■ small general agency) 
Oregon (a snail general agency) 
Michigan (a medium sized general agency) 
Virginia (a medium sized general agency) ' 
Pennsylvania (a large general agency) 
Mississippi (an agency for the blind) 

The objectives for the Model Evaluation Units (MEUs) were tn- 
work for ~J:.l4Xl:^^^^^^ net- 

iZTi.T.s': '"'"^''^ °" a„r?:s':?n:!tS*iSo.TLiu- 

requlrL'^^'lhaLlirst'^ar^s^ ^'"^ Evaluation Units were 

Dy their first year s contract to perform the following tasks: 

1. Plan and organize the Model , Evaluation Units' 

Initiate and establish continuing working relation-- ' ' 
ships with associated organizations, contractors 
ana university resources' 

3. Administratively monitor 'the organization of the 
Evaluation Unit- 

4. Field Tost the new (revised) Federal Program 
Evaluation Standards- 

5. Field Test the Vocational/Medical Facilities 
Information System; , 

6. Develop New Evaluation Capacity; 

7. Assure dissemination and utilization products- 

8. Evaluaf.e the project, and ' • 

9. Submit yearly and final reports. 



On October 1, 1971, RSA, after competitive bidding, awarded a con« 
Ct-act to tba West Virginia Research ^and Training Center (WVRTC) to 
coordinate the activities. -of MEU development. The primary functional 
responalbilities of the OTRTC are to provide coordination, promote tech- 
nical assistance^ monitor activities, develop models, conduct evaluations, 
and prepare articles and other materials for dissemination- The specific 
tasks of the for the most part coincide with those of the MEUs; 

however^ additional tasks Include the development of a regional office 
(RSA) model for the use of evaluation data generated by State VR agencies'* 
and the development of a set of manuscripts that will be Instructive to 
other agencies who want to incorporate MEU concepts/products into their 
' prograin evaluation units. 

Two other contracts have been awarded by RSA to provide specialized 
assistance for Tasks 4 and 5, Task 4, to pretest the New (Revised) 
Federal Program Evaluation Scandards, will be facilitated by Berkeley 
Planning Associates (BPA) , BPA developed the new standards under a 
previous contract with RSA, Their involvement with the MEUs includes the 
pretesting and refinement of -the proposed performance, procedural, and 
project standards. Activities will include designing instruments for 
data gathering, training the States in instrument use, coordinating the 
pretest J analyzing the dataj revising standards as necessary, and prepai-- 
ing an implementation plan* 

Walker Associates, (WA) , under an agreement with the National Associ- 
ation of Rehabilitation Facilities s will be providing specialized assist- 
ance to the Model Evaluation Units in the conduct of Task^S - Pretest the 
Facilities Information System 0; IS) . The FIS was developed by WA under 
a previous contract with RSA. The major tasks that WA will be conducting 
include training for MEUs in the use of the FIS, pretest implementation 
assistance, monitoring and evaluation of the piBti^sC experience, revision 
pf the systein where necessary, and the development of recommendations for 
nationwide implementation* ^ 

The reports of this project will contain (a) issue papers that ral^'B 1 
questions and suggest answers in generic evaluation problems; (b) des- 
criptive reports of the methods employed and results of particular evalu- 
ation studies; (c) "how to do it" articles; and (d) reports documenting 
the experience of the various staffs. These reports will be published 
quarterly for a period of three (3) years. 

This, second issuance of obBarvatT^ ns contains an outline supported 
by flow charts and sunmini s of Di lawaiu'y case review process* Oregon 
deployment of staff in tracking: Issues is described and illusLrated wttli 
an eKample of the technlqv l „ork. An interim report from Michigan 
examines the program evali ^r's role as consultant through two case 
histories* Virginia's appi ich to evaluation uses team techniques employ- 
ing both evaluation staff a. : program personnel. Pennsylvania's contrl= 
butipn describes the structure of its case review process in terms of 
regional and district roles. West Virginia describes an ongoing study of 
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lllilT^ mt^ °' using- benafit^cost analysis In vocational rehabill- 

s1sJen*miLrie f d.valoptnent of a management information 

s>scem might well precede attempts at evaluation, 

^ Number 2 in our series concludas with a review, contributed by West 
Virginia J of a valuable new handbook in human sBrvices evaluation — which 
will join our bibliDgraphy of pertinent literature. 



Richard A. Nida 
Project Ofticer 
December 1980 
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INTRODUCTION AND CONCLUSIONS • ' 

ona if ajocatlonal Rehabilitation counselor is a mny-facetcd 

activities i^tfJi '^ ^T""' '° ''""^ functions'^nd 
^i,Li\icies into two broad categories. 

stan da"uL"bVF.d»:r'r' ' Job are 

senerallv '""=1 -'ate regulations and procedures. These are 

It IS expected that most of thesa are resolved throuer 1 • 

over time. reaoiVed through , l application 

JHl_2gtal - A large segment of tha counselor a activities could be 
individuals "o" -^J^tlve as the services provided are prov.'^d to 
maivxduals. One cannot standardize the actions and r*=-,^r-f^„ - ■ 
beings and the individual skills and know! d of'he c"un e or^in 
handling the specific problems of . client will always be iost sl^lfi- 



cant 



Scm^lJ^^J^?°^°^°^ counselor activities, the Case Review 
bchedule (CRS) applies primarily to the "Technical or Procedural"" 

prSrobJec^Lesf '"^'^'^ °' " ^ ^^^^^ ^^ree 

regS'Sranrs^idSJLs: '''' ^—"^^ ^^^^ -^-al 

2. to be utilized as a supervisory tool for feedback to rehabili- 
tation counselors on case documentation and case practices, and 

3. to be used as a management tool in the determination of rerom- 
pracSes!- "-^^"^-^^^^ ^ service documentatio^ a^d 



THE CASE REVIEW SCHEDULE (CRS) PROCESS IN DELAWARE ' 

After-Delaware DVR first learned- of the San Diego .Case Review ' 
Mavl97fi ",«f°8^«° Evaluation Conference held in Syracuse. N ^. in 
May 1978 the Agency decided to investigate the use of this instru;kt' 
to evaluate compliance with Federal regulations and guidelines! A^an 



1 
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aid to the counseling staff, the CRS process subsequently denonstrated 
its merit for training and helping delineate responsibilities of the 
vocational rehabilitation counselar and supervisor. Later, after DVR 
was awarded the ME/MI contract, a second use for the CRS process was 
determined. It was identified as a feasible instrument for gathering 
data for Task IV. of the Model Evaluation Unit cnntr act—pretesting of 
the new Secretary's Geneial Standards. ^\ 

From the outset, the Agency has consldereci the CRS a management tool 
and as a procedure monitoring devlce^^not as a method to evaluate indivi-- 
duar counselor performance. 

Thus far,,,Jelaware DVR has provided CRS training to key adminlstra'^ 
tlve staff, all casework supervisors, and all counselors, thereby becom^ 
ing the only State Agency, tu date, to provide its counselors with such 
training in the use of this nationwide, standardized case review Instru- 
ment. San Diego State University CRS staff has been the sole trainer and 
consultant throughout the CRS process in Delaware, 

Analysis of the data reviewed in Fiscal Year 1979 has shown that the 
Agency is doing an acceptable Job In meeting the requirements set forth 
by the Federal regulations. It Is hoped that- when-additioTtai-data- beco 
available, improvements will be realized in those areas where weaknesses 
have been Identified. These improvements would be due to the fact that 
the total staff has been trained in using the CRS. 

\ ■ 

The impact of the CRS on the Agency decision-making process h:-^ been 
substantial* The original twelve revlawers (represented by five cuunsel-= 
ors, four casework supervisors, and thtee Central Office staff) became 
members of the CRS Committee selected by Agency administration to re- 
conmiend s plan of action with respect to the findings that resulted from 
the CRS review. At the memento the approved reconmendations are either 
planned with definite dates for implementation or havealready been 
implemented. (See Table 1.) The Casework Manual has been revised, 
procedures have been clarified, training has taken place to correct 
weaknesses, and priorities as they relate to these items have been 
established. The. CRS has proved to be an excellent tool to identify 
strengths and weaknesses within the casework process. " 



TECHNICAL DISCUSSION 

Two required basic concepts outlined by the CRS Program of San 
Diego F-tate University have been considered in preparing for Delaware's 
CRS applicntion on_a continuous basis* ^ 

A. Random Sampling : The application of the CRS is designed 
t J take a representative sample o£ selected cases from the 
State VR Agency as the CRS is not designed to be used on 
every case processed through the VR system^ 



B. Reliability I The reliability design has been bullc into 
the application of tha CRS so that consistency amoL re! 

JJkiSlinll nfSd! ^'"^"'"^ ^"^^ ^""^ 

The CRS application process can be presented in four steps. 

1 m Plaiining < 

2. Review ' \ ^ 

3* -taalysis of Data ^ 

V. Follow-Up % 

ftas provided technlral a^g^e^^^ < j ^ - ^^roject btatt 

procLsing of datf SpSfLSl t "sponslbillty for the 

had tolaO doLsiSlfSglJdinsr"";'^^^"' administration 

A. The sample characteristic fojr the selection o^ rases 1 e 
open or closed casea. Close'd cases (Statuses '26 and'28^ 
recoinmended by-the CRS Cotolttee for FY-80 review! 

'atS'of tt^t ^^^Yie^^s. bRS Committed reconMended utili- 
zation of the 12 or^igmal reviewers. 



D. 
E. 



The-^requency of the review, i.e. monthly or quarterly.' 
Length of time for the compilation of" data, i.e. 6 or 12 months 
F*' Target date. ^ 

^^^^J variables changes as the Agency's needs and reouiremeiits 
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■analysis of data This phase 
esisaclally takss plice mt 
San Dltg© State University 
uadef the aoordination of tht 
P&E Umlt whi£h Is also 
raepemsible for the diasemina* 
tion df-. the results whieh leads 
to the F&44^w*Up pha.se. 
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^* FOLLOW-UP^ This final 
sctp coisaaneig uich ch# 
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POLLOW-UP 



f Oil, DVR ^ 



DisEf ibutts 
£ tchnleal 
report 
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I VR stiff 
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4(e) 
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will bfi eValu/.Etd with 
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4(e) 



/ Ind of ^ 



1 K 



As 



indicated earlier ^ the Case Review Schedule has yielded a number 
of posijtive reconimendatlons. These have resulted in important changes, 
and have initiated a number of actions which will result in long-range 
improvements, ■ The CRS Cbmmittee has reviewed all of tlie ^recoiranendations 
and those selected for further action aTfe suntinarised in TaBle^l* 

As for FY-1980, a ^Ix-month period review was begun in April and 
will continue through September 1980. We are hoping that when CRS' results 
become available at the end of the year improvements will be realized in 
those areas where weaknesses , were Indicated. Accordingly ^ a determina^ 
tlon will be made as to whether additional training measures are needed* 
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piagnbs t le 
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DE^WARE'S Ihn^LRMBNTATl PROCEDURES OF SELECTED CRS RECOMMENDATIONS 



ACCEPTED RECOMMENDATJONS 



ADMiNIST^TIVEr 



(1) 



Add confidentiality and release of information state^ 
ments to DVR-:3 application form (from IWRP Form 6): 

"All Inforjnation pertainlnf 
to my cafe and documentation and other 
pertlnenr material will be kept con- 
fidential and will be released only 
when necessary in developing my ritabili- 
tation program or upon written consent 
by me." ' 

(ir; v Revlse^DyR^J (Application for Services) to include an 
r: explanation of^the Client Assistance Project (Item 4c) 



III 

(iiittihiiit*)a) 



(2) 



(3) 



ERIC 



VR Group Responsible 
For Implementation 



Forms ConimltCei 



Reinforce the inclusion of the case record of "Basic 
.information" gathered during the Preliminary Diagnostic 
Study as defined m the Delaware Division of Vocational 
Rehabilitation Policy Manual Cltems a and 6). ; ^ 

^f9"?r?,?'^^^ ^he case npte entry which reflects the 
eligibility determination include the cburiselor ^s 
analysis of factors creating the mih^^nnti^i l.^n^^^^^ 
to employment ritomg k anH Kj ^ — ^ 

The Casewor k Manual and Forms " Committee should develop 
document ation criteria arid methodology that will 
adequately satisfy state and federal rMuiremonts 
(Items B and 60. ''■'-^^^ 



Forms OomialtteV 



CRS Skill Training 



CRS Skill Trniiilng 



Caiewqrk Nartunl 
Committee 



(CONT4. 



1 ( 

cm ' 



Acceptccl 
Rccomjnciulnt ions 



IV . 
Feasibility 
Eval, of 
Rehab* Poten 
tial 



ADMINISTRATIVE^ 



<4) 



(2 1 




ERIC 



Provide an expanded definitiDn of: '^related fnctors" 
for the Dqlawnre State agency, using tlioso listed in 
the Rehabiljttation Services Manual 1505.03b as a 
reference: _ 
These factors may include,: but are not limited to 
the following* 



a. 
b, 
c * 

d. 

f. 



lack of marketable skills; 
low ndunntional lev^I; 

community and employer prejudges and nttltudDS 
concernifig disabilicy: i 
long-term unemploymciit; 
unstable work record; arul, 
poor attitudes toward work, family, and 
community. (Ref. 



2 7 ^ r:ne g Rgvlow Manual ) 



/ 



Expand dste gatheriilq techniques to establish 
crjj^eria. tor tho evaiuation of factors which 
bea;; on the cliont*.H handicap to employment 
(Item 5). [Factors such as Medical Psvchologlcal , 
V^ational, Rducatjonal § other related) fRef, n. 25 
Casp Pc view Manual ) 



Bsta&^Jish guldeliiieit procedures, and minimum criteria 
^'rr coynself^r annlyHis of the nine variables covered 
by Items nmphasls ^boUlcl ^be placed^ on how to 

t7rit0 anJ donumeiit briafly an* clearly. (Ref. p. 11 
iim mnmi) Jl^v^ tf.vgrlables to be descrlbed/appriiised: 

Piim o-jiisUt achievements. 



"IS 



Vli Group Ilesponsible 
VoT Implementation 



Casework Manual 
Committee 



GWMComm, -Staff Dev. 
Of f leer-Casework 
Supervisors 



GWM Gomm, in conoulta^ 
tion CW SuperviBOrs and 
District Admlniseratora 



(com. 



cm 

Sect ion 



XWRP SEatus 
42 and Above 



Acccptccl 
Hocommenclat loiiB 



ADMJNlSTRATIVn; 



(2) 



(3) 



Ability to acquire occupational skills 
Capacity for succGssful job porfornianco . 
Client^s employment opportunities V 

Bxpand or enforce current chapter on Similnr ncncfits 
in Delaware DVA\Policy Manual. (RevlGw Information^ 
available in the\ publication on Similar Benofits 
issuod by the Research and Training Center at Stout 
Vocational Rehabilitation rnstituto) (Item 3b) 



Revise IWRP Form 9 to include "None" next to 
"Similar Beneflts"l 

Attach dollar valui to Similar Benefits which 
indicated on IWRPSi 



' re 



(4) Increase supervisory seijutiny of limp's In Regions 
2 5 3 to assure Inclusion of progress toward 
employment goal (Item 18), 

(5) Revise IWRP Form 4 to include the indication of 
periodic and annual review • 

(6) Refine IIVRP Form 4 to assure that the specific 
objective criteria by; which the progress of the 
client Is evaluated documented (Item 19), 

(7) Determine annual review date (from .date eiient 
entered Status 12) utilising program computer 
print-out (Item 22)1 ^ " 



VR Croup Rq^pbnsibie 
For Implemontatlon 



Casework Hanual 
Comml t tec 



Forms Committee 



Medlca^l Fees Commlttae, 
Similar Berief Its Coord 



Forms Cominlttr^^ 
Forms Committee 



EDP Soreenlng Commlttet 



(CONT.):, 



ERIC 



ID 



CRS 

Section 



Accpptocl 

R b c 0 mm 0 n d a t i o n s 



. VI ^ 
Delivery of 
Servtcee 



VII 
Te rni Ina t ion 
of 
Cases 



ADMINISTRATIVH: 

(B) Incease suporvlsory scrutiny of IWRP Form 6, 
enforcing the requlrQmcnt of the recording 
of the client's views of the program, If eliont 
does not state his/her views, record "None" 
(Item 23) . 

(9) Review the state policy and amend, if necesHary, 
to assure that thq Certificate of Ineligibility ' 
(IWRP Form 1) Is issued in all cases closGd in 
atatus 30 or 28, and that proper documentation is 
/ recorded oji the certificate according to RSM 

, 1549. 03c and CFR 130i:37c (Item 24c I Item 28) 

(10) Revise IWRP Form 1 to include client's views of 
.the Ineligibility decision to meet the require-^ 
ments for CFR 1361, 39el, 



L) Reinforce need for documentatidn and evaluation of 
similar benefits pertaining to all services 
(items Sg and 7g) 

(2) Require more extensive provision of Placement 
Services " 



(1) Increase supervisory scrutiny of cases to be closed 
due to loss of client contact to assure substantial 
recontact efforts (Item 31b) (Region 1). 



VR Group ResponsiblB 

For Impl emonta tlon 



CaBework Manual Comm. 
(Revise ,Item 10,*24i of 
CaBewor^' Manual ^ p*62) 



Casework Manual Comfn. 
(Policy Issue) 



GonCingent: ij , ' d eel si© 
re A^f^r^ j: ' n^' on 1 

sue ui cer if :ica te of 
Ineligibility 



CRS Skill Training/ 
Casework SupervlBors 



Casework Supervisors 



TABLE 1 feont.) 



CRS 

Soction 



Accepted 
Recommdnda t ion 



ADMINISTRATORS: x \ / 

W Increaso superv^isory seriitiny of closure 
activities (Statuses 30 F, 28) rolatlve to 
tho eligibility decision to assurb that 
when posf.ible: / = 

a) full consultation with/the client (or 
client's representative) was made 
(Item 35, also refer /to CRS Instruction 
Manual, page 52) , and 

bj the client was informed in writing of 
the closure action^ taken Cltem 36).'^ 

(3) Increase supervispry scrutiny to assure the 
prosence of necessary documentation in closure 
lettep to clients and/oi/ closure statement 
as iv/Ri amendments to be/given to clients 
(Item 48) , / . ^' ' 

(4) Increase supervisory review in Region 3 to 
assure appropriate reporting of closure" 
outcome to other agenfcies (Item 41) and adjust- 
ment closure form to/ reflect that the appro-, 
priate agencies were contacted regarding 
closure action (rtfer to Casework Manaul 
Committee) * / 

(5) Revise IWRp- 7, wl^^n stock is exhausted, to 
include job ti^I^ as well as the code 
for the type of/occupation (ftem 42b), 



VR Group Responsible 
For Implementat inn 



Casework Supervisors 



Casewerk Supervlsora 



Casework Manual Comm., 
Formg ,Comm,i Casework i 
Supervlspra 



Forms Committefe 



(coNT . y 



TABLE 1 Cconto 



cRs: 

Section 



III 

Eligibility 



IV 

■ eaelblllty 
^val^ of Re 
lab Potent- 
al 



Accepted 
Recommend at ions 



TRAINING; 



(1) 
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CI) Contingent upon the results of reviews subsequent to 
the initiation of Administrative Recommendri tlons 
1, 2, and 3, develop t raining techniques to assist 
counselors in clocumentatiDn ot clotn synthosis re- 
latlve^ to eligibility detorinlnation (Items S and 6). 

(2) Train counselors in recording the results of inter- 
views bases on Information to be gathered as defined 
Xn the Delaware Policy Manual (Items 5 and 0)/ 



Provide training and develop skills In diagnostic 
interviewing, analyEing, and documentation with 
specific reinforcement of the definitions of 



^describe" and "appraise'^ as outlined 



on page 50 



in the CRS Manual of Instructions (Items 8-16), 

Contingent upon the results of future .reviews 
subsequent to the implementation of Admini- 
strative Recommendations 1 and 2, consider 
skill training in the synthesis of information 
in relation to the client and the vocational 
goBl with emphnsis on clarity and succinctness . 



VR Group Respcfnsible 
l ^nr Implomenta tlQn 



CRS Skill Training 



CRS Skill Training 



Staff Dev. Officer 



Staff .pev» Officer^ 




TABLE 1 (cont,) 



CRS 

Section 



V / 
IWRP- 
Status 12 
^Jid above 



VI 

Delivery 
of 

Services 



Accepted 
Recommendations 



TRAINING; 



CI) 



Provide training In the proper recording of tho 

tSlD-^f^^ similar benefit investlgcitlon of the 
IWRP [Item 3c) * 



C2) Train counselors on the proper procedure for 
completing revised IWRP Form 4, 

(3) Train counselors In procedures for evaluation 
ot s^ervice delivery in terms of ouCeome 
(Items 18-21), 

(4) Provide training in documenting objective 

, criteria, procedures, schedules, and results 

fioS^^i^^^^ progress evaluation on the refined 
IWRP (Items 18-21). 

(5) Provide training In documenting satisfactory 
vocational adjustment on Status 26 closure 
amendment (Item 25d). 



(1) ^^/ Provide training In documentation and evaluation 
of Similar benefits as they apply to ill (Items , 
Sg and 7g) services* 

(2) Provide tralnlny in the provision and documenta- 
tion of Placement Services. Use the definition 
of placement activities ^s provided In the Rahabi» 
lltation SBrvices Manual 1S41.01 f-l), 1541 03f2V 
1541,04, and 1541*07. 



Vn uroup Responsible 
For Implementation 



Staff Dev, Officer 



Staff Dev, Offlci 



Staff Dev, Officer 



Staff Dev, Officer 



Staff Dev* Officer/ 
Similar Benefit Coord. 
(Trng, at Oet. meettngy 
District Stiff) 

Staff. Dev, Officer 



(CONT.) 



TABLE 1 (eontj 



CRS 

Section 



VII 
ermlna fe ion 



Aecc[^tod 
Recommendations 



TRAININGt ^ ; 

(1) Prpvide troinlng In developing suparvlsory con- 
sistency in writing adequate closure stateinont^ 
as IWRP amehdments and closure notification to 
the client (Item 48), 



C2) 



(3) 



Provide training on counselor documentation pertain- 
ing to placement services with "as appropr j'ate" 
in mind (Item 54a through i), 

Prdvdde training in the documentation of criteria 
related to suitable employment, specific to items 
SSb and c* (Refer to Administrative Recommendation 
number 9) , ^ 



VR Group Responsible 
For Implemontation 



Si-aff Dev* Officer & 
CaBework Supervisors 



Staff Dev* OffleeT 



Staff Dev. Officer 



(CONT. 
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TABLfi 1 (conclusion) 



EXANjPLES OF RBJIiCTRD CRS RRCOMNfENDATIONS 



CRS 

Section 



Rejected 
RecQfnmenda t i ons 



REJECTED- 



II.A 

Preliminary Dlag^ 
nostvc Study 



II. B 

Extended 
Evaluation 



IWRP Status 12. 
and above 



pSv^P^?''- -/i" regaraing use of State Agency 

Psychiatric/Psychologicol Medical consultants: 

: ■ \ ■ ■ ' ■ 

• conl^mnfs current status Df DVRis Medical 



1 



This CRS it_„. ... 
Federal Regulations 



tlons"°'' is not required for cotnplipn^q with 



ejected samplo, review M^^^ 

Out^of 120 cases complotioly reviewed, only 91 deceived exf^H^,! ■ 
ln1x?end2H rWbn,T„ended that a'sample of 30 ""s&t^,, 

and n^fS^- evaluation .be reviewad to determine areas of streWths 
and needs improvement. The number of extended evaluation cHeP^^ 
closed in statuses 26 and 28 is relatively small ?o be o? ^ ^ 

P foJfty^S aid irsL'id'd-" ?oca,,mendaMon' is a low/ 

areas " should direct our conceTiti toward other - 



Reject data pertain 
cases closed non 



to the'prpyislon of the annual review of 

, 'rehahilitated., PrSvision is made on JWRP 

Form 6. Reviewers, at the time of the^review. did ritft taL ini-n 

iSdicaJJ-?KP"J;*f^"" °^ ^^f ^^6- TSIrefo^e. aniyrfd yS - 
indicated that Delaware lacked such provisiorir \ / ' J 

account. ; The validity of the reviewers responSes'was questioSedf\ 

. ■ ■ ^ ^ . ■ . \ 



ERIC 
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: . ^.^^w.. t^^u .SYSIJM 

GEMLD V. MANN, ASSISTANT ADMIMISTRATOi 
FOR POLICY AND PROGRAM DEVELOP^Nt \ 
OREGON DEPARTMENT OP HUMAN RESOURCES ^ 



Oon^,^^^Z^^^ Evaluation Unit .as to davelop a 
Identified and analiad and ho^- oa^ ^ f >^ ^^'^ P°^^=y issues are 
and evaluated. SomK^ollcy decision^ ^^"^^^^^^ Mde. i„plen,ented, 
individual in tha orS^riLh l • * «i"8l^ 

At the bther end of the^oSnu^ 1^"^? Irtteraction with others, 

large ^Kpenditure of sLfrtime^f """"" which require a very 

Involve'ment Of a widelLietf of staff """"'^^ P«i°d- 

outside the aeencv arp - asency, and perhaps from 

data before a'dSkon ifraachld 'IvlrVf S"h-^,^tudy and'analy.e- 
been made, the amount of staf f tL Snd 

implementation can vary enomouslv^ r"^' "^"^"^ - 

Vocational R^ihabilltatlon Dlvlaiir mi f P^®'' program- such as the 
staritiaT^ber of such polS 1« ^^''^ " any given time a sub- ^ 

3 decision inaklngf or Slementati:"'' various , stages of analysis , 

be.made dellberata^: Suchl^LlLf « '""^""^ '^Wch should • 

o. e,nst.ntly shiftLg ^^^^ m^'m$^^my. 

Given the number of llsues thp ah^ft.' '' ^ . . 
. changes, and a general crisis orlen^J^ "^ priorities, the staffing 
. programs today it Is nor nn n * ^° "'^"y human service ■ 

losf, leavinr ^hi^dmlnlstraSrS ^ becoma 
^m^it is riquirad disldvantaged by not haying vital data 

would'^gJe^tS^ "^y^Lf L'°^^ administrator. 

impllaations. to conSouslJ a^ iSll^^ln^fL^^^^^I^^.P^l"^ 

Prppose alternatives and to complSa thf aetlo^ rJ^" such Issues and to 
i tion of the Administrator's decision t ^ through the Implementa- 
' Administrator with parSlc Sorts so thS should provide the 

necessary to insure that H^Hrf " changes^raay be, made when 

are focuLd on thl^^^ari;" .^-ra^JL^Me:"" ^ f^'T ' 

accountability. ^ ^ ' a'-tivinies. Such a system assures , 



"SWaeptual Framework. ^ accountability an adjunct to the 



to 



^^^^ 

recognition 4 K reacLd bf SldM " ^^^^^S^' This 

evolva fro« dlLusMon^ith others ^rlf'^^L ''"S'^' " 

^ .memorandums or routine management lata. ^ ' apparent^ from 

When the decision is. made to expend staff time anH . 
P^ticular iBsue, the following are requiredf resources on a 

1) Identification of the issue- 

2) Assignment and nqtlf ication'of a responsible lead person- ' 

Assignment, of an anticipated completLn date or thfdaS 
When a decision package is duei and ox tne Aat^ 

■ 'Notification of the person who maintains the Issues and 

Tdentifica,£io_n, and Tracking System. issues and 

LtSd avauL"?-^ on.such «t«rs a, when trdeclsia"- 

-salons Of Sister agencies. aS Jh^.'sLf f ^°J,e\"is"be°s t^Me'^f" 

- - - yur.wij.j. receive monthly progress reports* 
1980 t^^'f %^ff ^ in November. 1979 with 32 issues. - As uf May 31 

reoiilflT- ,-anfl^..^-4 s'^^ioiis required by higher authority but not includine 
tlfki^r — -"P^"^- "petitlve reports have their ow^ ^ 

tickler, system to insure that timelines are metr • , 

raaponSlllfcv'f'''' 23 different staff persons have been assigned 

three afanv Le°ff°"^ °l P^^on assigned more tSan 

to 18 months ' ^""^P^t^d -mpletlon time has varied from two 




Although a, compleK nimbering systam is currently in use^ we recoimiiend 
a simple sequential numbering of Issues be adopted for identification 
purposes i 

A word processing system is used to store the complete tracking 
systein. During the last few days of each month a reminder memorandum is 
'generated, using the word processor, for each issue being tracked. These 
memprAuda are then distributed to the persons responsible, who complete a 
handwritten progress report on each issue by the fifth working day of the 
following month. Only a few minutes are required for each person 
responsible to handwrite a concise progress report and route it to the 
one who 'compiles the monthly report for the Administrator. Likewise, the 
compiler exp^ends only a brief time co cheek each repart, underscore any 
delays or roadblocks identified, assign an appropriate progress symbol 
and route it to the word processing operatar for text editing. The 
monthly progress reports on each issue vary in length from a single 
sentence to as much as a half-page of single-spaced typing. Care is 
exercised to limit thy^ength of the progress report and include only 
that information reqiiired, by the Administrator for review purposes, 

Iri addition to the narrative paragraph on each issue ^ a quick 
reference inde^ and a transmittal memo are included to the Administrator 
to Increase the utility of the complete report. The transmittal memo is 
usee, to call attention to new issues, completed issues, and those which 
are not progressing as expected. The index also contains a progress 
symbol for each issue. The symbols arei "S - satisfactory | "PS" ^ 
pattially satisfactory or minor revisions are required; and^ "U" ^ un- 
satisfactory* The narrative paragraph for each issue which is identified 
ag unsatisfactory or^partially satisfactory contains underscored sections 
which describe the problem. This method of identification permits the 
Administrator and/or other appropriate supervisors to be readily aware 
of any problems and take whatever action is necessary to get the issue 
moving again. 

Monthly progress on the issues is rated satisfactDry for the majority 
of issues, and there are usually three to six issues which are considered 
only partially satisfactory and one or ^-'^^'o that are definitely unsatis- 
factory. Corrective action is taken ^ wnen appropriate ^ as a result of 
the tracking process. - / 

We believe this system provides a reasonable degree of accountabili^ 
ty with a minimum of staff time expended. It also provides an informa- 
tion flow to all appropriate staff on a variety of vital issues. Final- 
ly, the Administrator's Office is fully informed and can take action to 
shift priorities 5 resources, and staff in whatever way necessary to 
accomplish that which is most important. 

\ 

Copies of the June ^1980 Issues Report, together ^vlth the "assign- 
ment memorandum'" and the "monthly reporting memorandum," are attached. 
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r.^ l A essential to have such sophisticated equipment. The system 

could be, initiated on a manual basis with bnlv ^iJ^^L system 
tod typing time required. ^ slightly more secretarial 
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OREGON VOCATIMM. REH^ILITATION DIVISION 
* ISSUES ASSIGNf^T MEMORANDA" 



r- 

SUBJECT I VHD Immumm Tracking 

YQUshava baen aligned responsibility for a TOD issue on which 
the Admlniatratar has determined that staff time and resources 
will be expended. In order that the Administrator and others may 
be -kept Informed on the progress of this Issuei a report la re^ 
quired fr«i'yoii by the fifth wrking day of each month* The re- 
port should ooncisely deaorlbe tho^se aotlvities related to the 
Issue which were soropleted during the previous month* Ifhen ap- 
propriate, you should also reconunend ohanges in antlqipated oomi? 
pletion date or any other matters of which the Atainlstrator 
should be aware aonoernlng this issue, 

h consolidated report of progress on all Issues being tracked' 
will be prepared monthly and a copy will be supplied to eaeh re^ 
sponsible person^ Please retain this assignment for referenae« 

h reminder statement for each issue for which you are responsible 
will be sent to you near the end of each month and it should be 
used in Ipif sparing your monthly report* It need not be typed^ but 
please write legibly. 

The issue which has been assigned to you isi 

Humber Title Anticipated completion date 



Narrative description of assignment: 

(When appropriatei this description takes the form of a measure- 
able objective 0 
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TOs 



FROM I 



nnmoH vocational rehabilitation oivisiofj 

"KOfJTHLy REPORTING KEMOIWJDUM'' 



1. 
2. 



DATEi 



SUBJECT: VRD Issues Accountability Report 

You have bean assigned responsibiiity for the following: iasuei ' 

Numhor Title «..•■.', 

— ' — — — , . Anticipated completion date 



Last month's progresa on'this Issue was rated __ 

P^**" progress on this issue dur- 

ing the current month. Return this report to me no later than 
the fifth working day of next month. Pleas« identify any prob- 
lems which have caused or may cause a delay in completing work on 
cms issue and any recoramendations for change. 

Progress; report: ' ■ 
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STATE OPvOREGON 



NTEROFFICE MEMO 



TO Administrator*! Office 

Dale Reeves^ Harv Schubotha 



PROM Program Ivaluatiort k Statisties 
Ntil Sherwoed 

SUBJECT P^oqcmss Report for May 1980 on VED^ Issues and Action Plan 



In aococdanoe with your memarandum of 11-30-79* tha progress report 
for April' is herewith submitted, ^ ' 

This report covers. a total of 30 issues* one of which has Just been 
added. The new issuM is entitled, "Study of the Younger Nursing Home 
Population to Dete^minie Feasibility for Moving Sot© to Lais Costly 
Care* Report Results." It is assigned to Mr, Stephens* 

Two laaues have b^an completed with this ra^rt. They are Issue 
Number € - pevelop an Equitable Sheltered l^orkshop Pee System* and 
Issue Number 17 - Completion of 0^ 1977-7S liennial Report, These 
isauas Will not be carried on future reports. 

Seven issues are coniidered to be pr^ressihg only partially satisfac- 
torily, because of the -reasoni underlined in the narrative portion of 
the report. Two issues are conaidered to be progressing in an unsa- 
tisfactory manner. They are Issue Number 9*5 * To Develop Operational 
Plan with LMal Schools* and Issue Number 9-6 * To Develop a Crants 
Management Handbook, The problem with both appears to be that work 
considered to be higher in priority is crowding out any time for 
action on these iasues* 

Twenty issues are progressing satisfactorily and one carries no prog- 
ress rating as it has just been assigned. 



Id 

Attachments 



Qdi Bob Butler 
Sue Druffel 
filsie Forrest 
Irene Grahim 
Ton Huff smith ' 
Laufilee Matcher 
T,C. Jamas 
John Jelden 
Gerry Mann 
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Sandy Hat thews 
Ross Horan 
Clarenee Persad 
Jerry Rmpel 
Ken Russell 
John ichreiber 
Clyde Stephens 
Doyle Taylor 
R^l, Wright ^ 



32 



ERIC 



INDEX 



2 
3 



5-1-d 



5-2 



S-3 



S-4 



S-5 



-ISSUE OR gsjECTlVi: TITLE 
Services to Injurad Workers 

.Jnd€pendent Living 

Implementation of Si 5SSS 

Bttmnqthmn vm'm Information 
System a Evaluatipn Capaoity 

Rtviae and Dooument a Concip-* 
tual Framtwork for Informa* 
tion Flow and Poiioy Dtvelop- 
fa#nt within VRD, VPubliah 
Document. 

Plan and Conduct Field Teats 
of the Fropogad Standardi 
for VH ^rogrM and Project 
Evaluations, Report Findinga, 

Update vro»s Capacity to 
Enfage in Program Evaluation 
Aetivitita by Providing , 
Training to' Projeot Staff and 
Others. Report Results, 

Develop a iyatem for fraoking 
Major laaueg under Study 
within tmo. Rtport Heiulta* ' 

Develop an Improved Forras and 
Reoorda Management System* 
Doeufflent Reaulta. 

Field .Test the Faoilities 
Information System* 

Evaluate the Costa and Bene* 
fits of VB Servicea 

Study Waya to Reduoe the 
Duration of VR Services to 
Clients 



AHTICIPATED 
CQMPLOTldN DATS 

June 1980 
^ {revised) 

December 1980 

June 1981 i 

Septaraber 1980 

June IfflO 
(reviaed) 



Auguijt 1980 



July liao 



December 1980 



June 1980 
(revised) 



June 1981 



June 1§30 



June 1980 



PERSON 
RSSPONSlBLg FROaRESS* 



John Jelden 



Evaluate Population Esti- 
mates i Rtsourge Alloeatien 
Models^ 

Evaluate Sffectivenesa of 
Job .Placement Service i , 

25 



July 1980 
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Heil SheMQOd 



Robin Wright 



PS 



fde Stephens s 
Jerry Rem^l q 
Nail Sherwood s 



PS 



Roaa Moran 



Laufilee latcher s 



Clarence Per sad s 
(revised} 



PS 



Laurilee Hatcher S 
(revised) 

Ross Koran b 



Irene Orahao 



September 1180 Rosa Moran 



Doyle Taylor 



m^ssR isgtJE OR dsJEeriVE title 



ANTICIPATED PEHSOH 
CQMPLSTIQH DATE RESPONSIBLE PRMR fiSS* 



9-1 



9*5 



12 



Develop in Squi table Shel- 
tered VlQctcshdp Fee. Syitem 

Develop Weckshop Cectifica-* 
tian Standard MonitQcing 
Syiteffl 

Monitor and Evaluata Shel- 
tered Serviees Subsidy 
program 

.\ 

Bvaluata Mental Health/ 
Deaf Projaot 

Develop Operational Plan with 
Local Sohooli 

Develop Granti Kanagement 
Handbook 

Review and Revile Client 
Caseload Procedures 



May 1980 - John Schceiber 

(eompleted 5/80) 



June 1981 



March 1980 



March 1980 



June 1980 



John Schreiber 



January 1981 John Schrelfaer 



January 1981 Elsie Forrest 



Irene Grah^ 
tcro Huffkmith 



Terry Janes 



12*5 Recast ^he Adtainistrafeive July 1980 

Manual 

14-1 Develop Peraonnel and Train- 
ing Information System -> 
Flans " 

14*2 Develop Quality Accounting July 1980 

and Purchasing Services ' 

14-3a Train Staff of uaera Guide, 
for MIS ^ 

14*3c Dtvelop IndeKed Repository 
of Frofessional/Technical 
Information (Phases I i II) 

14-4a Develop Supervisory Training July 1980 
on Dealing with Problem 
Smployas 



Terry James 



PS 



[to be determined) Sandy Matthews s 

Ken Russell S 

August 1980 Sandy Matthews S 

July liiO Nell Sherwood S 



Sue Druffel 



Pi 



. _ ANTICIFATSD PERSON 

wmm ISSUE 21 objective title cowlctioh dms resmhsible 

14-5a Impltraant a Revised AutMated Junt 1981 Bob Sutler' 

Fiieal System 

17 CMpletion of OTOD If 77*7a May 1980 . Robin Weight 
Brannial Report ioomplttad S/80) 

18 Study the Youngar Nursing Sept, 1980 Clyde Stephens 
(ntw) Hamm Fepulation tu □etermine 

Feaiibility foe ttoving Soma 
to Less Costly Care. Report 
Results, 



FROGRESS ^ 
S 

PS 



* S * Satisfaetory 

PS * Partially Satlsfaotory - se# underlinad seotion of report 
a * Uniatisfaotory 
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Iisu« Nun^ar 1 - Services to Injured Workers - 

'Ac. Jelden caports that* substantial effort was expended during May to 
develop a draft of the contract between HCD and vm. It was not L«Lle 
, it IM m anticipated eg mpletlon d,te ^l^^Tig^^ 

Issue Number 2 - Independent Living - , / . 

reports that there has been no racvement on this issue 
federal >^n. T the approval of Its grant appUoation ?L 

MA hla h H*?/"- 'w*'^* ^ 9"ne progra« in Oregon. Approval In 

Se ofd^f - f; °' ^'^^ ne« Eduction Dep«t«ent and 

f ^«le9«tion of authority to the MA Comissioner. The antici- 
pated oorapletion date of Dec. 1980 remains. 

Capacitr-^" ^"^ -.Strengthen VRD's Information System and Evaluation 

Mr. . Sherwood reports that progress as adequate except for somB tf^la^ in 
documenting th4 iapaot Of the recent . nialsKative re«gani«Sn tl lL 
H^T^ »fdel for information flow and policy development (See issue 
r^PH ^ ^^^'i' ^" documenting an improved forms anf 

recoraa management syatera (see laaue HiuBber S-l-e) . 

«atlon''^%^d^;"J Document . Conceptual Pramework for infor- 

mation Plow and policy Developraent w.i thin. VfiO. Publish Document ' 
De-ao"".i ; .,^5""'^ cepocts a conulnuation of the procesa of realigning' 
perse . al and functions within the recently established ocganiaational 
fehl "f^^ «««^««: this and other i^rkload has resulted in delay of 

^^^if^^^^neeptual framework document required by the federal oontLcJ 
The completion of this document is now targeted for the end of "unl 

^t^*^f«"r^*^ -"^"^ ■ Conduct Field Tests of the Proposed Standards 

foe VR program and Project Evaluations. Report Findings T " ^^"^"-^ 
f sports that data for the closure and 12-month follow-up sur- 
veys hw been collected and entered on a computer t«oe. Copies of tWs 

dSa Si be f^'f?'*' ''J coordinating contractors The 

fta f loyally analyied during June and a report wlU/fae distributed 

th. fJn^ f fff; * follo,^up lurvey wiirbe mailed to eacJ of 

the 300 rehabllltanu who were sent a closure survey. This component of 
the overall study will ccmn.ftce in July and run through Octoberf 

Icti'vi?^*^ 5-l-c - update VSD'a Capacity to Engage in Program Evaluation 
a«ilt" '"'hiding Training to ProjMt Staff and Others. Report 

Ml. Hatcher reports that Mr. Wright continued his forms management 

f*^*' training was received during toy. Ma. Hatcher began work 
on tne development ^of a workbook with specific TO examples for teaching 
Eaaytrieve programaing within the agency. ceacnmg 

ilthin"S^' 1"^"^" ^^'^fi^P * system for. Tracking Major Issues under Study 
witnin vWt Report Results, - 

eaports that the issues tracking system is in place and ffunc- 
of "he'laarSf'*'?* b^i^S ^iven to Issulnl a remindfr K 

2™J Lf^^ P«"«n responsible for an issue. 

Hopefully this procedure would result in a more timely lubmiasion of the 
monthly ce^rts, 

apprecla^el'**"°"' impcovement of the Issues Tracking System will be 

slhift!^?"*"!^""'' ^.^^ P«s«<3 will assume reipon- 

siblity for the preparation of the monthly issues report. " 
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fepocts that a draft of Che system documentation is being 
prepared for staff review. Coroletion is nr^ . ° -"^ 



Nftlon^rr"^"^"- ~ ^i*" '"ting of the Proposed 

■ National FacUitie. informatian System. • Document Plana. 

M«r,^5l f?r"r*^"* ^^'^ training aesaions for "^Oregon rehabilita- 

^i^f " during May. One was held in Portland a^d 

i ^ h-""'- facilities «ere represented. Mr. Schr'?ber 

- field te-ris'^cherri'' *f f ' i*"^" °' ^ Thraotual 
r^ionalhiM^ "'^•^"i*^ " begin oA July 1. Ms. Hatcher will aaau«e 
responsibility for this issue in -.the future. 

Issue Number S-2 - Evaluate the Costs and Benefits of VR Services 

/'^ f^" "P°"» that he has oomplated.a draft cost/benefit model 

IJlnalK". L^"?"'f J"' -'itical review, wdj' 

tionally, a copy is, being reviewed by Dr. Richard Dodson of lerkeley pan- 
ning Associates. Dr. Dodson is an author ity on benefit/cost analysis orvR 
?s reL^e^def "^'^^^ «ltlois«. ynless substantial revisit 

il ' ''lli b« coapleted on schedule. Once consensus 

ncr^^-i^^H °" default assumptions of themodel, it will be possible to 
ihauif L automated con^ponents of the procedure. This revlsefproceLre 
should be tested with the FY 80 data available in October. -P^o^e^"" 

Clients'^'" " ""^ " ^* Duration of VH Services to 

ta thi'J^f 'S^!" * qu«8tionnalee is being prepared to distribute 

' rLv^I with Mr? 

Reeves. Suggestions will be sought as to ways the field could visualize 
carrying out thta assignment. ^ visualize 

iodeL^^*' ■ Evaluate Population latinates and Resource Allocation 

Dr. Mor an reports that estimates of nunber served and case service 
coots were made through FY 83 for agency priority groups. He wiUbe 

atlon"Snlf "^^^^ °' "^^ram Planning and Evalu- 

ation Dnit to develop predictive resource allocation models. 

issue Hunber 5-5 - Evaluate the if f activenesa of job Placement Services - 
o^ th^iS^i" f*^"^ -^"^ Collow-up on tn* issue of increasing efficiency 
o£ the Job developers, which was raised as a consequence of earlier progrS 
evaluation studies. Is now essentially complete. In addition to a balil ' 

ded ln"2h?o™^^' developers, the following change, have been recSen- 
ded in the program model, and endorsed by the Atolnistrator .• 

1) Wherever practical, job developers will bm aasigned to a aiwU ' 
office, under the aupervislon of the office supervisor, it is ore- 
aupposed that the greatest "teM- efficiency la realised when all 
raepbers of the team are accountable to the sane team leader. 
To maximize job developer skills and aervlcea, only those clients 
wlth^ the more severe handicaps requiring mote time or extra effort 
in the placement process will be referred to the job developer 
3) Placement efforts shall be carried out In accordance with an indi- 
vidualized, written placement plan mutually arrived at by 'the coun- 
^? if^*^^®"r developer. Progress on plan implementation 

Shall be moaitored and reeordsd in the client file. 
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Issue Nujaber S-5 (continued) 

4) There Wlii be tn increased' -eehab' expectation from taan afforts 
which include job developers. 

S} The basic worfcplan will allow for individual diffecencas in aasign- 
raents of job developers to accoraraodate the unique circumstances of 
the assignment, 

5) Aa^ vacancies occur in the Eraploynent Specialist secies, th.se posi- 
tions will be converted to VR Counselor I positions. Counselors 
ni.wd into these positions will function as placement counselors' 
for the first year of their job asslgnaenta. 

• Caments, suggestions and input on the coomon elements of the job deve- 

loper workplan have , been received from raanagera, supervisors, , and job deve- 
lopers and will be incorporated in a revised draft model. 

Issue Number 6 - Develop an Equitable Sheltered Workshop Fee Systen - 

Mr, Schraiber reports that the workshop fee aodel has been completed 
and accepted by the workshops In the Association (OARF) . This issue has 
. now been completed and It wiU not be necessary to report further on It. 

issue Number 7 - Develop Workshop Certification Standard ftonitoring System - 
^ "F; J«*>«^lb«^®Poets that The Workshop Association has agreed that vm 
adopt, CAW acereditatipn with the. costs to be shared equally for the ini- 
tial survey by the faellity and the Division. 

JShf Administrative Rule to mandate CMF and to adopt the revised stan- 
^ dards has been submitted to the Secretary of state for adoption on July 1, 

Issue Niaaber 8 - Monifeer and Evaluate Sheltered .Workshop Subsidy Program - 
Mr. Schrelber reports that part of the baseline data - client wages - 
was received and the averafe computed for each facility, information 
regarding clle« terminations, client dlaabilities. and refer r -^l aourees 
has net yet been received from AP3. ' . ' ~ ~ 

Issue Number 9-1 - ivaluate the Mental Health/Deaf Project - 

Ma. Fosrest reports, that the Interim Evaluation Re^rt on this project 
was conpleted and distributed to ail interested parties In Oregon during 
April 1910. Copies werer.also distributed to the itate Coordinators for the 
Deaf In other states as well as to the central and regional offices of HSA 
Several connoendatory letters have been received from recipients. 
A final evaluation report will be prepared in January 1981. 

Issue Humber 9-S - Develop Operational Plan with Local Schools - 

Ma. arahaa reports that copies, of the local plan have bean diatribufied 

to the field. Req^aests, have been made through Regional Managers for signed 

documejitg. Sye regions simply don' t want to do it. 

It is suggested that we reinitiate this process"in the field with a 30- 

day dBadline for eompliance. 

l33u« Hiaiber - D«v«iep dranta Manageraent Handbook - 

^ Mr. Hyffsmith r«p©f ts that the evef-endurlng Granti Managtment Ewdb^k 
has been ;typ«d in raugh draft ferol 

Mmwmvmwp as a raiult of r#e©m#ndatlona madt by aur fid^ral fciend3r 
Clyde Sttphani has roughed out a leotion on II,R to bm included in the Hand- 
book. KlBQ, the fedeifal friends suggMtad we beef up the sectiona dealing 
with evaiuatlon, eto. rinally, the Hand^k needs to be' revised to reflect 
changes made as a result of the most reoent aAalnl^tritivt reofganliatlon. 
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lisue Humb« i*g Continued) ' 

atiof Indiwtis that after getting the Planning *nd S^Uu- 
atlon unit ocganlied as far as Job duties are oonMrntd. -^thia project will 
atlf^ of^h '° P«"«n «Bpl«ion. Until, then.^in'consider- 

«larlt^ °^ business, it *.iU go on the shelf as a very low 

issue Number 12 - Review and Revise Client Caseload Pfocedures - * 

d^lavfi reports that the final case developbent forms have been 

•^ablLhT^In P^i°f"i"- " ^ther new priorities are n« 

established, the draft foraa will be printed in June. 

Jun.. "^^"^ * scheduled to be distributed to all Manual holders in 
and thrSSstraS:'' '""^^ "^^ 

Issue Number 13-5 - Recast the Actainistrative Manual - 

Mr. Jaaes reports that n»at of the volumes schedules of the Manual 
revision project are on schedule except StaM Oevelopm ent and Tralni...i 
fjfhjre ge rfformance to .late \w ^mn unsatiifactorv. — ^ 

tssue Number 14-1 - Personnel and Training Tracking Information System - 
the ra'lool Jl^L'^-tr^ I*" Pointing have slowed progress on. 

b^ uSiiJSh f t f : ^""^ instructions have been received and will 
M Utilized beginning in June. 

Training Dnit staff reviewed iJtBcutive Department's remote Access 
System which will b. used after July l, isao. This ayitem will be used as 
the personnel data base for both Training Unit and Personnel Unit 
d-,iff ^ I'Twf^ u"" ^« developed before preliminary ays teas 

d-slgn and establishment of a meaningful completion date. ^ ^ 

issue rubber 14-2 - Develop Quality Accounting and Purchasing Services - 
Mr. Russell reports thati 

1. amploye Travel and Purchasing of Client Services and Supplies OVRD^ 
Manual aectiona were issued this month. 

2. Inventory of Adainlatrative office fixed assets was completed 

3. Continued work on new Q9m Manual. 

4. Participated in liB Workshop Review of procedures (aceounting } by 
Region X. 

5. Attended the accounting portion of a Facilities Horkahoo in Port- 
land. - 

f . Survived a General Services review of VRD purchasing policies 

7. Revised mail roan schedule. ' 

8. Met with Bob Butler and Mike Mcflride on the •MARS" system 

9. Mike Mcaride started work on ■MAHS." 

Issue Number 14-3-a - Train Staff on Uaers Guide for HIS - 

Ms. Matthews reports that the needs survey is completed and the data 
has been analyzed. While changes in the system could be proposed fiscal 
™«HfJ?"?f " BOdifications requiring repyogranmlng or report 

modifications. Due to this, a training package wiU be developed awund 
management reports currently in existence. ' 

Issue Number l4-3-o - Develop an Indexed Repository of Prof asiional/Tech- 
nieal Information ^Phases I and II) - 

P^. f'>;f«ood reports that the target date, of Jul> I remalna attainably 
for completion of the initial phases of the development plan. 
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Issue nv^ms - Develop Supervisory Training on Dealinf with >roblem 

Ms, Druffel reporcs that dye to the decrease of one staff membec in the 
Personntl Unit and inertas^d workload involving day-to-day personnel iisues 
and problems, ther# has not been any recent work accomplished in developing - 
a training program for our supervisors* She does not anticipate any time 

^v^ilable jor this pcoltct in the near future □nder the present Ttaf- 
iinq allocation, - _ _ ^ " - " 

i£ M gegomroindaeion to assign this project to the Training Unit " 
^^if^g Hs. Scuff el as an advisor or consultant, — 

t^%um Number 14-S-a - Implement a Revised Automated Fiscal System - 

Mr. Mclride reports that the. general design of the revised system is in 

progress, it is too early in the design phase to identify specific tasks 

whicn could be scheduleU. 

A detailed schedule v#ill produced soon after July 1, 1980, as the 

fiscal input is applied in more detail. 
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Issue Number 17 - Complefelon of Q\mD»s If 77-78 Biennial Report - 

Mr, Wright reports that the report was distributed to selected staff in 
early June, An evaluation questionnaire accompanied the reports and except 
for analysis of questionnaire results this issue is completed. ^ 

Issue Htwiber 13 - Study the Younger Nursing Home Population to Determine 
Feasibility of Moving to Less Costly Cart. Report Results. - 

This issue was assigned to Mr, Stephens as of June 2 so action will be 
reported in the nesct report. 
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THE PROGRAM 1VAI.DAT0R AS EVALUATION CONSULTANT i 

TWO EPISODES "~ ~~ 

' ' .. ROBERT H. RICHARDSON, Ph.D. ' ; .J 

< ; PROGRAM EVALUATION UNIT ' / 

■ MICpOM BUREAU OF REHABILITATION 

The futidamental task of the program evaluator is to generate accurate 
^J'"^ f'''^^^^°u P^°i'«"« «ld to commi^^d that infor^aSoJ^ 

in a usable form so that, managers can make th* decisions for which they are 
responsible The key descriptors in. this task definition are "useful"- 
• ?ta utlllJ^'f f ^^««ft?l characteristic of a "good" evaluation is 
Its utility to decision ihakers. However, any program evaluation can only 

ln.n^""i"i f u'^^^^^" as the adequacy ^of "the evaluation design 

Incorporated in the^ project from its irvception.- lfr there is n. design for 
evaluation or If the program lacks specific objectives or falls to artlc" 
win ^^"f^^^M whlih program outcomes are^ to be measured, the design 

or isabl^"?^f^""f-^"^ the "suiting evaluation will fall to produce useful 
or usabJLe information* ■ . r \ 

conM«;;«M "'f situation^ proposals for .lew programs or requests for the 
continuation of existing, programs contain a detailed evaluation section 
specifying the littended outcomes of the program and identifying the specl- 
IdLr^J^f V^-^ process and outcome evaluation. Again, In an 

Ideal situation, the program evaluator would be responsible for routinely 
reviewing the oevaluation section of program proposals and for certifying ■ 
their evaluabillty in terms of the Intentions- and expectations of manage- 
ment. Those proposals and requests falling; to obtain approval in this 
process would be designated for remediation In consultation with the uro- 
gram evaluator. iThus, Ideally, a routine process df Irevlew and consulta- 
tion would guarantee both the evaluabUit^ of agency programs and projects 
and the utility of the Information generated by program evaluations. 

In most situations, however, the work of ,the program evaliiator takps 
place under conditions which fall somewhat short of thi ideal. Many pro- ^ 
grams are considered to be valuable by definition and are never required, 
to ^ndergo formal, evaluation. Other programs are put. together so hur-rled™ 
ly because of proposal deadlines or other reasons that there is little* 
expectatien that either the stated objectives will be attained or that th- 
evaluation design will be Implemented as written. ^ It la often only latter"" 
in the event of a financial shortfall or ^i^^ecollectlon of an Impending' 
report, that the issue of pro|ram^je4;alSgBiiity becomes Imniedlately relevant 
When this Is the case^jih«r^ to be dona? What can the manager or program 
evaluator do to satisfy the manager's need for useful and usable informa- 
tion for decision making? One solution, although less than Ideal, \is the 
utilization of the program evaluator as a consultant to design an evalua- 
tion midcourse which, is fair and equitable on the one hand,' and which 
satisfies the manager's need for accurate and useful information on the 
other. 
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The purpose of this article is to describe two episodes In a state 
vocational rehabilitation agency in which the program'evaluator .'as g"en 
this assignment. Both consultations are still in process at tWs wrltlne 
and it is not knoim how they will turn out. In addition to the in'-ended 
outcomes of the consultation, all the participants are gaining new insight 
with respect to the impor.tance of evaluation design and of having an ade- 
quate desr^gn in place" from the beginning of the program. However the 
program revaluation unit is also learning that it needs to be prepared to 
provide assistance when that condition does not exist. ' ' 

Episode fi l j 

■ " . - ' 

For the past three years, one of the agency's local district offices 
, has participated in an RSA=funded facilities staffing project with a local 
medical facility specializing in physical restoration services for the 
severely disabled. The notion underlying the project was that, given the 
size of the hospital and the number of patients who became clients of the ' 
surrounding vocational rehabilitation district offices, it would be mutual- 
ly beneficial for both the hospital and the state agency to place a full- 
time vocational rehabilitation counselor in the hospital to coordinate thP 
delivery of renabilltatlon services to patients from within the jurisdict^ 
of the adjacent local district office and to serve as liaison between the 
hospital and outlying district offices for the delivery of rehabilitation 
services. This pilot project proved to be so successful— "mutually satis- 
tying might be a more appropriate term since the original goals and 
objectives of the project lacked objective criteria of success and depended 
primarily upon the subjective assessment of the principal parties— that 
both thf hospital and the agency desired to continue the relationship 
This desire was consumated with the signing of a match agreement to take 
effect on October 1, 1979. ' 

On December 27, 1979, almost three months after the initiation of the 
new agreement, the agency program consultant responsible for monitoring 
the agreement, met with this evaluator to lay the groundwork for a plan 
for evaluating the project. The program consultant presented an histori- 
cal overview pf the project^and described the close working relationship 
which had evolved between the agency and the hospital. When asked about 
the, goals and objectives of the project, he indicated that they were listed 
m the match agreement and that I would receive a. copy for my review. In 
response to the evaluator 's repeated emphasis upon the importance of highly 
specific objectives with measurable outcome criteria, he Indicated that 
although the objectives listed in the match agreement were "somewhat 
general," they enjoyed a high degree of consensus which had developed over 
the three years of the pilot project. . "The problem," he said (and this 
was the first mention of anything problematic in the relationship), "is not 
su much with the goals and objectives as with the responsibilities of the 
counselor t,o the hospital on the one hand, and to the vocational rehabili- 
tation agency on the other." 



".h ^^^^f^>y^^ description of "the- ptotilen"; prompted one of those 
e?the 3f recognition on the part of the evaltrntor which reorient- 

all ^«^..^it*^''""^°" ^""^ ultimately clarified the nature of the 
evaluation task. It was,, after all, "the problem," not some abstract ■ 
interest in program evaluation, that provided the primary motivation for ' 
lem ""L?^h t , f °8^*" «^^l^«or. Furthermore, the nature of "the prob- 
lem, which had to do with the delivery system through which the project 
thit'^It'J'" ff^T' -ith.the obJectlves'.s such. IndlcaJ^d 

■ atlont . H 2 ^^^^8" °* * Product or outcome evalu- 

ation but a design for process evaluation-a system for monitoring and 
evaluating the processes by which the program products would be delivered. 

These insights resulted in a series of meetings convened by the' agency 
program^consultant and , involving the project's hospltai=based cLsupervisoJ 
agency-based co-suppviaor, and the program evaluator. This working group 
made a thorough review of the stated goals and objectives of the agreement 
and the criteria by which these goals and objectives would be evaluated 
soeclfl^^^n^ evaluator was never satisfied tha. they were sufficiently 
bS n ■ ^ f "^ adequate outcome evaluation. However, in view of the 
broad consensus In support of the project goals, and in the interest of 
dealing with the problem which prompted the request for the evaluation the 
evaluator heeded the ancient adaget "If it al^t broke, d^'t fS lt ") 
^addition the group articulated a set of four basic process objectives 
which, together with their sub-objectives and activities, specify the pro- 

At t^ls writing hf' f ""^^'f i-Pl--ting her'assignaS t^skte, 

^4 ^' ! co-supervisors of the project are installing a data 
collection system that will enable them to monitor the status of each pro- 
corL^Jn" "^.^^ ""^^ those programmatic adjustments and course 

nra^r^ T*, ? ""'^^ ""^"^^ effectiveness and efficiency of the 
throuLVhrr % ?° d^S'^^^ion of designing an impact evaluation 

through the installation of a follow-up of hospital-agency clients. 

Episode #2 , . 

Prni Jl'^^rS^^? " * concerned with the evaluation of a Client Assistance | 
Propct (CAP) in a state vocational rehabilitation agency. The circum- ' ^ 
stahces surrounding the initiation of the evaluation consultation were ' 
soriewhat fortuitous. Technically, this f ederally-f unded proj ect had been 
in existence for two years and was applying for ah extension grant that 
wodld become effective at the end of the third (current) fiscal year. 
Actually the project had a very unstable .beginning, characterized by a 
sl|-raonth delay in funding, philosophical differences between the staff 
and management with respect to implementation of the project and ultimately ' 
a change in leadership three months into the second fiscal year. The first . 
tasks of the new director were to write a first-year annual report, plan 
the program for the second fiscal year and recruit a staff. Under these 
circumstances, it Is not particularly surprising that, the project director 
Who had had no previous experience with program evaluation, included a 
rather weak evaluation design In her project plwi 

, ■ \ ■ ^ ■ ■ 
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It was at this point that om of the, senior managers of the aBencv 
approached the program evaluation unit with a request that someone Be" 
assigned to help the CAP director design an adequate evaluation. Manase- 
rpent anticipated an extremely tlght^ financial situation for the next fiscal 
year anu would require concrete evaluative; Infomation to facilitate de- 
cisions which would have to be made about the projecb should It fall to / 
receive a federally-funded extension granti Furthermore, the mkager ln'^1- 
cated that he wanted ^concrete Inforniatloniabout the contributions of tie 
project to the aiency- what would be missed If we didn't have It (the/ 
project), how this kind of contact with clients results in somethin/dlf fer- 
ent for clients and 'system-changes' which Improve sarvJces for clfints." 

At this same time, the program evaluation unit. Itself par/of a' 
tederally-funded project of the Rehabilitation -Services Administration 
whiph was designed to create midel evaluation units in six st2tes was in' 
the process of selecting agency projects 1^ which to fleld-t4st a set of 
project^evaluatlon standards developed by,4nother Federal dontractor. What 

. thought, , to being together our/ own emerging 

skills with an eitplicit need on the parfi'of management afid some specific 
resources developed by a nationally-known contractor.' /Thus, the decision 
was made to enter into a consultative/ relationship wiih the CAP director. 

i / 

The first meeting of the evali^Ator and CAP director was a friendly 
disaster. The evaluator learned a- lot about the CAP project and the CAP 
director learned a, lot about progtam evaluation.- But things began to fall 
apart when the discussion turned to an evaluation design for the CAP pro- 
ject. As a first step, the evaluator «nd the CAP director reviewed the 
legislation and regulations underlying the national program and the evalu- 
ation section of the CAP proposal ot this igency. The CAP proposal essen- 
tially mirrored the' statements of purposeHncluded In the Federal regula- 
D ^"fJi ^'®'' "To increase client knowledge and understanding of Bureau of 
RehablliCation services and processes, of client rights and responsibili- 
ties, of appeals mechanisms, of civil rights and of . other public benefit 
programs.' The problem is that this Is a process statement— what fhe staff 
will do—not a product statement— what the staff .will accomplish. Recall- 
ing the specificity of the agency manager, the evaluator continued to press 
for statements of product, outcome and accomplishment. For example f'What 
will happen as r. result of this activity?" "What will the outcomes be?" 
Invariably,, the CAP director's response was a restatement of the original 
'jrocess. objective, "Clients will know Bureau of Rehabilitation services 
and processes..." Ajid equally Invariably, the evaluator 's response was. 

But how will we know that they know?" "What will they know?" "How many " 
will know?" Then came the explosionr "You are forcing me into a position 
of declaring, in quantitative terms» what the specific outcomes of this 
project will be.' I don't know what the outcomes will be. We have no 
history or experience to guide us yet. I was under the impression that 
the agency believed that these services were Important in their own right." 

The CAP director was absolutely cdrrect, of course, and the evaluator 
felt duly chastened for exerting the pressure that he did under these cir- 
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would .„abie-.hL to'Sj°diL^Lir l^°oii:;dSrrti°^\^^^ 

' of actJviSi""T"eaa°'acM-°r? '"^ '^'^ ^^aged t„ a number 

obvious that -very aativlrJ h J " ""^ contrary. It was rather 
As a^,«tter of St as S f dj^r '° ""^"'^ or .expected outcome, 

tl.s tf h.„.«i . ; J director began to describe specific aotlvl- 

wouldn' t Jt bf, '"^ '° anticipated outcomes! M^y 

:o~ pS^^^^^^^ 

make a li^r of ^,11' " session. The CAP dlractor and her staff would 
^er ?he course oJ P"Jf ""vities^-everythlng in which they were engaged 

outcome of e"h ^ft^vi. Tl ^^^^^ '^e purpose and expected 

ouccome or each activity m mind, wa would group the activities -fn t-o™ p 

ti^"with'"" or outco.es. This process IhouLlLu" l^T e of objecf 
tlvea with .nown or anticipated outcomes which could be monitored lur the 
purpose of outcome evaluation. Admittedly, these obJective^Mght r. fleet 

I^tenJed fr^olo'^But "1'^ 'T' -^^"^^^ -^at^gency mSgemen 'haS 
thi ^, . J? ? ' S^""®" uncertain beginnings of the project and 

Set StJer thifrh" ' is-""" of management Intentions forihe prof 

? - ^ prpcess=llke statements Included in the enabllL 

legislation and regulations, this would be a start==a base for negoJXtine 
with^management wlth^respect to oth^r or additional expectations fnd o""' 

this poSfllhe'cA^^f '^"P'" °' ^Pi^^de cannot be written at 

this point... Tne CAP directoy and her sfcAff have identified llterallv 
scores of focused activities. Keeping S intended o^tcomL of these 
fied1i"functionai ^ -^^^ program evaluator have identic 

beltated in tie for^'^of ekch\f^ich has the potential to 

be stated m the form of an objective with specific measurable outcomes 
The resistance to this type of. specif icity is inevitably present giveh the 
lack of a track record for the project and the fact ^that no other tao^ 1 

sneclflM^ ''f f ^ ^^""'^ ^"^^ ^ '""del with respect^olhe 
back lito L design. The temptation is alway' to slip 

difficult ^^'''v f °f statements which sound so good, but which are so 
airricuit to evaluate, * 

steps will entail first, presenting the evaluation design to 
agency management for approval or negotiation of additional progranmatic 
objectives. Then, being sure of a set of criteria for euranativf e^luation. 
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the CAP director and program evaluator will complete the deaign of an oper- 
atlotial tracking system to measure the progress of the project toward its 
objectives and a data retrieval system to ensure the availability of accur- 
ate and useful information for both formative and summative evaluation* 

The episodes reported In this article have portrayed the program, evalu- 
ator in the role of evaluation .consultant. This role is not entirely new 
to most program tvaluators, of course* However, In most Instances, ideal- 
ly, the consultation takes place on the "front-end" of a program rather 
than in the middle. It is to be hoped that as managers become more aware 
of the utility of evaluative information for the decisions they have to 
make and as program administrators become more experienced in designing 
adequate evaluation into their programs from the beginning, the need for mid 
course evaluation GOnsultants will decrease. That should be a happier 
siutation fpr all concerned* 

As indicated above, neither of the coniultatlons r^eported in this 
article are complete at this writing. The author/consultant will be happy \ 
to share a descriptive report of the fruits of either or both with those 
who are interested when they are completed. 
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' ^THODOLOGY FOR DISCRETE EVALUATION; . 

JANET SLIPOW, BILL BROWNPllLD, JACK HAYEK AND BEV KAUFFMAN 
. PROGRAM EVALUATION SECTION , ' 

VIRGINIA DEPART^NT OP REHABILITATIVE SERVICES 

Reh«h???.^^?*"? Evaluation Section (PES) of the Virginia Department of 
Rehabilitative Services (DRS) has developed an evalultion mthodology L 

g^nerL pS'lff "f ^"-^^"^'"f The methodology speeifl^f the' 

fHreLntedtni HI - ^""f 'fr' these , internal evaluations and will 
be presented and discussed with prograB mnagers during the prellminarv 
evaluation activities. The purpose of the methodology " is to promote the 
proper development of meahingful Information for decisionmaker" 

1^ defines evaluation as the "process of delineating, obtaining ana- 

Stu?flebeari973?' "rfl ^f"-"- J^dslng decision alternaf ives" 
activities 'rn B; S««if, advocates two goals for all evaluation 

J ? - ^"e"ive, useful and timely information sharing among the 

Sevel^^^''^' f f ^ evaluation activity, and (2) technical accurlcy in't^e 
development, implementation, and reporting of the evaluation actlvLy" 

a consequence, six basic knowledges. and strategies are highlighted 
throughout the evaluation methodology. PES will strive to maintain them 
during every step of the evaluation activity. The following list specif ies 
these on-going knowledge' and serateglesi I " ' speciiies 



(1) 



Uhderstand the policy making, process of the organization. 



(2) Involve appropriate Information users and audiences in the evalu- 
ation activities, ' , . 

(3) Employ cdnsulting skills throughout formal and informal sessions. 

(4) Provide accurate information In a timely, relevant and applicalbe 

Adapt the evaluation activity to meet changing Information needs. 

Develop and maintain a. mutual, problem-solving approach. ■ 

mth these factors in mind, an oyerview of. the methodology reveals Its ' 
four stages: focussing, planning. Implementing^ assessing, and rennrtino. 
Focussing establishes the scope and obj ectlves^o f the evaluation activitv 
and emphasizes the active, early Involvement of program participants 
Planning produces the blueprint for conductitig the evaluation activity and 
prepares for the dissemination' of results arid recommendations. Implement- 
ing obtains .the necessary information, analyzes and interprets findings 
and provides results and -alternatives to the decision makers. Assessing 
and rep or ting provides time for decision making, and a raetaevaluation (a 



(5) 
(6) 
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DRS metaevaluatton Includes an evaluation oif the evaluation effort, input, 
process, outcooie, and the use of the resultant inf ornation) . Eventually,' 
an appropriate impact evaluation may evolve from this activity. Each of' ' 
the four components are interdependent and a feedback loop exists among all 
four of the stages. A graphic portrayal of the overview follows: 



FOCUSSING 



^ PLAKNING 



LOTLEMENTING 



7^ 



ASSESSING 

Mm. 

REPORTING 



J 



eedback 
op 



Each stage of the evaluation methodology Is further specified In the 
following see ticns of , this articled A graphic portrayal of each stage is 
included with the destription. In addition, a final scheviiatic, inclusive 
of all elements of thk process, is presented at the md of the narrative. 
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£01 w! " , "as, 6£ the evaluation 




will bTsZ^^la f \ rationale and purpose of the activity 

.cey sL^nera'on -in'^hr o'te're'lfor * ^t^' " 

,ldentl£les appropriate staf£ pe;a"^a"i„ i''p°^?;ic:;a%':j::™;j?n":£fort., 
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Once managersV needs' and purposes are Vlarlf led, an important question 
must be answered, " Is evaluation approprlat^ r' At this time an analysis of 
the type of information necessary to satisfy the manager's purpose is made. 
Representative Cs) from PES and the appropriate\nmnager (s) discuss the 
area(s) of qoncern and provide applicable Inforimtlon* In these early 
developmental meeitlngs key issues will be clarified. The Project Evalua- 
tion Team may be r.omposed of PES staff appointed By the^ Program Evaluation 
Sectuton and program personnel designated by the un^: participating in the 
evaluation. The availability of program Information, dnd rights of conf^.- 
dentiallty will be discussed. In addition , the practical implications of 
decisions based on alternative evaluation outcomfes wi^ be acknowledged and 
any bias and/or assumptions about the evaluation will b\ specified, PES 
emphasizes the effect of impressions developed during initial meetings and 
considers the development of a mutual, problem-solving reMtionship as be- 
ginning with the first encounter among the case of characti^s. As a conse- 
quence, during this and subsequent activities, conmunicationK to promote 
clarification and understanding is stressed. The terms djsci^s and tiagotl- 
ate and satisfactory ? Indicate this communicative, problem-solVing strategy. 
If no agreement is reached and an impasse cannot be resolved, then the 
evaluation is stopped. If consensus is reached, the evaluation proceeds to 
the next focusing/decision making activity, . ^ 

An evaluation agreement serves to provide a framework for the practi-- 
cal administrative tasks of the evaluation. With agreement, the cast of 
characters .in the evaluation know general eKpectations and are able to. plan 
appropriately with these in mind. PES names this declaion block of the 
mfethodology. Agreement Setting: Are Resources Available? This agreemsnt 
setting activity includes a discussion and negotiation of eatimated costs, 
staff identification and definition and the allocated time frames for the 
completion of the activity. The time frames for completion of the activity 
focus on the managers* needs for the results and practical considerations 
for the conclusion of the activity. Again, should an Impasse be irresolv- 
able, then the evaluation activity would cease. 

The agreement partlcuiarly emphasizes the eKpllclt creaticn of ^ the 
Project Evaluation Team. The Team conducts the evaluation, and is^^'revi- 
ously mentioned, 1^ composfed of st'.aff persons from PES and from the program 
being evaluated. The role definitions for each person will be designated, 
such that PES and programmtic personnel will understand their particiflar 
duties. These individuals function as a team from this point until the 
conclusion of the evaluation, shairing technical responsibilities, trouble- 
shooting any potential difficulties, and presenting conclusions and re-^ 
commendations to the specified audiences. . . 

Once .resources are assessed and the agreement is set, the newly formed 
Project Evaluation Team researches the program to obtain necessary back- " 
ground information . An analysis of this Infortnatlon evolves into the 
setting of evaluation objectives* The precise evaluation objectives are 
reviewed with managers for concurrence. Should disagreement be Irreconcil- 
able, the Project Team returns to the background dnformation to rS'-analyze 
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the context of the evaluation and nianagers are solicited again fo. 



ur es I 



^The next phase, the planning process, contains the following proced= 



i t DEVILOP £VALUAn^DESlCN| 



/ develop evaluation ?LAn^ 



I /REVIEW EVALUATIi^^n 
1/ PLAN Aim DESIGN 
WITH HANAGERS 




MODIFY ACHiHIiT^TION OF 
THE PLAN WITH ^UNAGERJi 



> 



^^^^^ 



H.^ni Plannini^Process Involves the development of the evaluat ion 
|MiE. During this phase each evaluation objective is reconstructld as 
an evaluation questlon(s). The purpose of the question Is to direct the 
eno »hf of information. Consequently, It must be specif Jc 
enought to contain "measurable concepts which can be investigated systeL 
atically, yet general enough to elicit information which is Lanlngful to 
decision makers- (Yavorsky, 1977, p. 59). The design should conJafnlhe 

f^f J ^""^ evaluatlon^^questlon (how the information wl if 

benefit the decision makers). It should designate the sources of the 

conectaH "i" P'^^^"" instrument, how Information will be 

collected and the type of instrument used to collect the information) 

1^77 p; 57 62)'' i"f"'"«ion is needed should be specif led (Yavorsky, 



The Evaluation Plan is the administrative schedule of the activity ' 

JatLart^f n? P"""" action steps for the Implementation. 

Categories of information to be considered are: the activity the evalii 
ation question addressed, the dates of the activity, Teamlembers coord^- 
atlng and participating in the activity and other, participants in the 
activity (Modified from Yavdtsky, 1977, p. 79). For the Idmlnistratlon of 
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Instruments, 1/ the categories should include instrument status (e;|. on 
hand, to be developed) evaluation queation addriasedj administration" 
schedule, adnlnistrator(s) , respondents, sampling procedure, data analysis 
procedure, data analysis, and date reports completed. (Yavorsky, 1977), 

In addition, the Evaluation Plan Includes a dissemination and utiliza- 
tion component. This designates who will receive what Information, ho^ ^ ■ ' 
(formally, Informally), in w.^t form, (pral presentation, executive suhmar^; 
written report) and at what point during the evaluation activity. The 
metaevaluatlon is also scheduled at this time. '. 

• When the design and plan are completed, the Project Evaluation team 
and managers meet to review the plan and to resolve any admlhlstrative " 
scheduling difficulties which may arise. Should agreement be unattainable, 
then the evaluation is stopped and a report is presented. 

After the planning is completed and managers are in agreement with the 
administrative works,, the evaluation activity enters the actlon-oriented, 
implementing phase. A flow chart representing these activities follows- 



Ij It is important to note that the development .of instruments would 
demand an Individual administrative schedule addressing such pertinent 
concerns as Identification of coordinating staff persons, .development of 
variables, reliability, validity, pretesting, any modifications, etc. 
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EXf CUTl EVM.irATION PLAN 
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1 RESULTS AND PRILIHINARY 
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f ■ \ ■ 


PRESENT FINALI2ED R£itn.tS AITD ^ALTERNATIVE 
. RECOf^INDATIONS 
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The Proj^t Evaluation Team executes the Evaluation Plan ■■■mp1^m»n<- 
' mg the actual data collection and analysis stages of the evaluation 
activity. When the analysis is complete the Project Evaluation Team 
develops the results by interpreting the data analyses. An answe^r to the 
question, ^^Has the Evaluation oeen conducted In accord^ee with technical 
standards?" Is a process check to inaure technical accuracy in the pro- 
cedures. Technical accuracy is examined by relating the evaluation 
results to the evaluation objecttives and considering the extent to which 
the objectives are met. The t^chnipal accuracy may also be checked for 
bias or faultiness by examinin|- the sources of Information, sampling 
procedures, instrument reliablkty and validity, analysis methods, or any 
other technical procedures. If the technical standards are found to be 
upheld, then the evaluatlbn process continues and the Team prepares pre- 
liminary recommandationo . If the standards have been violated, then the 
Team will identify reasons for the inaccuracies, and subsequently will 
discuss the feasible options. Options may be to (1) stop the evaluation; 
(2) return to the focussing phase and refocus the activity with manageri- 
al input- (3) return to the planning phase and replan and reconduct the 
data collection effort or (4) note the inaccuracies and go on with the . 
evaluation process. These inaccuracies would be reported with results 
of the evaluation (Draft and Final Report). 



After the technical accuracy has b'een determined, the Team prepares 
the results and p reliminary, alternative recommendations . These are 
reviewed with managers. The concept of discussing draft findings with 
the managers and decisionmakers Is one critical aspect of the methodology. 
Managers may hai^e additional reconnnendations to suggest, 



s^nagers may comment in writing about the evaluation. 
Lll be included in the final written report. 



In addition. 
These comments 



Finalized results and alternative reconmendations are presented to 
,the pdiences. Emphasis is placed on timely , usable , and accurate 
■information provided in a manner which is understandable and 



for the audience. 



appropriate 



The final presentation of the results and recommendations mrks the 
end of> the implemen ting s taae of the methodology. 

Assessing and reporting , the fourth and last phase, is summarized 
graphically on the next page. . 
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EXECUTE Rl^^INING 
DISSr-u-NATION MD . 
UTILIZATION ACTIVITIES 
INCLUDING A ^ffiTAEVALUATION 



FEEDBACK METAEVALUATION. INTO' 
PLAirarNG FOR FUTURE EVALUATION 
ACTIVITIES . 



Decision making allocates time for managemant to determine what actions 
to take regarding the evaliiation activity. The remaining dissemination and 
utilization activities will ba^gKaeu^flri based on the previously designed 
-schedule. One feedbaclc qyastlonnalre, the mfitaevaluatlon, will be adminis- 
tered to key staff, it will include questions on the information, usage 
and the Input, process, and outcome of the evaluation activity. 

Should the decision makers choose to implement the recommendations 
from the evaluation activity, PES staff will be available for tachnlcal 
assistance In these activitiies. IJhan appropriate, an Impact evaluation 
will be made to follow-up on the evaluation activity. 

, Finally, the results from the mataevaluatlon instrument and manager- 
ial comments will b6 fed back Into the proper planning for future eyalu- . 
fltion activities . PES striven to support the changing infarmat Ion needs'" 
of decisionmakers. To uphold this ideal the Section plans to change, to 
revise 'and to grow appropriately, ' 
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CA^ REVIEW PROCESS IN PROGRAtI EVALUATION 



HARRY W. GUISE. ADMINISTRATOR OF EVALUATION 
PENNSYLVANIA BUMAU OF i VOCATIONAL HEHABILITATION 

We In Pennsylvania f»el that In developing a comprehensive program 
evaluation plan, a case review process should be cne program methodology 
. in evaluation activities. A case review system has the capability c>f iden- 
tifying base line case service patterns, thereby generating Information 
tor the initiation of or improvement of decision making in policv and 
procedure, ; - , 

The extent and degree o| the structure and activities of the case 
review process, is contingent. upon the personnel " allocated by the state to ' 
evaluation and the needs of the state as seen fay top management and other 
inputs. , . 

■ Because of the proposed Federal Evaluation Standards which are current 

ly being field tested by the Model Evaluation Unit states each state in the 
country will have to develop its own case review process or utilize an 
existing process developed by someone else. ^ ■ 

To define program evaluatloji on a "state level, it is very useful to 
Identify procedures for utilization of information to Improve decision 
making in vocational rehabilitation program planning, monitoring and te=- 
vision. (Handbook of Program Evaluation Studies, Michigan Rehabilitation 
Research, 1978) This review of current state rehabilitation agency program 
e^yaluatlon studies further states that program evaluation studies can be 
divided into three broad areas: (a) input studies which provide Informa- 
tjion for US4S in program planning! (b) process studies which provide in- 
formation for use in program monitoring; and (c) outcome studies, which 
provide information for use In program' revision or change. In this i 
article our attention will be devoted to the process studies in program^ 
evaluation studies. 

A process study such as a case review system is concerned with the 
case service patterns within the rehabilitation' process. Examples of the 
type of information gathered from a case review system might b'si (a) 
delayed movement of clients' cas.es in th- rehabilitation process i (b) per= 
.centages of eligibility of served clients- and (c) suitability of selfection 
of clients vocational objectives. This type of compiled data would 
provide the appropriate people in the agency's case service, policy and 
planning sections a "nitty-gritty" information base for change, modifica- 
tion, development and implementation of policy and procedure. 

For example, we can determine through statistical data the existing 
time frames in the movement of clients' cases from referred status to 
eligibility status. If we develop a standard of three months as an accept- 
able time frame for that movement we might learn that in 20 percent of bur 




cases chere is delayed movement. With this statistical, inforniatlon as a 
base, a case review could be conducted on the. 20=percent-delayed^.Dvement= 
cases CO determine reasons fcr the delay. An analysis of the case review 
findings would provide sufficient case service intormatlon to implement 
case service policy changes, modifications or clarifications. v 

Above all, a case review process should attempt to meet the specific 
needs of the individual VR aiency uslnfc- the process. Tn Pennsylvania we 
teel,that we have a structured laso review process that does meet our ' 
particular needs and allows for input from. many levels. 

The structured process begins with out field operations organization, 
ihe agency organization' provides for four regional offices, each super- 
vising the field activities of several district offices. A case service 
evaluator is assigned to each of these regional offices. But though these 
regional ca.e service evaluators are assigned to the regional administra- 
tors, the central office Administrator of Ivaluation maintains contiriudus 
and direct contact with the four to coordinate and direct case service 
evaluation activities. This dual supervision and control assures a 
thorough integration of staff and line activities in program evaluation and 
Pi-ovides for the continuous flow of information to and from field actlvitiei 

The attached diagram (Figure 1) describes the relations^iips between ' 
the Central Office, Regional Office and District organization and the case 
review process. Utilizing this organizational structure, Pennsylvania's 
Case Review Process has the follot^ing ohjectlvea: 

1. Evaluate documentation foii stf '. and Pederal compliance. 

2. Identify the strong and weak areas in case service patterns. 

3. Evaluate the undersitandlng of existing policy and procedure. 
■ 4. Identify inadequate policy and procedure. 

,5. Standardize, as much as possible, the interpretation and imple- 
mentation of policy and procedure. i 

' 6. Promote a positive concept of evaluation's role within the 
vocational rehabilitation process context. 
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,With these objectives in mind, the following system has been developed 
to implement the Case Review Process In Program Evaluation in Pennsylv?.nla. 
This system has been further enhanced by Pennsylvania being a model unit 
state. Depending upon the Case Review and the circumstances, the steps = 
outlined can be modified, (Figure 2 is a synthesis of the Case Review 
Process. ) 

I, -. Selection by Office of Administrator for Evaluation of Case Service 
Area for Review 

A. Obtain input from a representative sampling of District Office 
personnel conceming Case Service areas that might be reviewed 
and the objectives for. such review; District Office personnel 
might be District Administrator, Assistant District Adminls^ 
tratttr and supervisors. 

B* Obtain input from Central Office Management concerning the Case 
Service areas that might be reviewed and the objectives and 
purposes for suggested reviews. ~- \ 

C, Through the above input crystallize and select a Case Service 
area for review and riCate the objectives of the review. 

II* Development of Case Review Sheet and Standards 

A. Each Regional Case Service Evaluator develops a list of approprl" 
atj questions co be used. for the Intended Case Review^ and a 
copy of each list Is sent to each of the other Case Service 
Evaluators and to the Administrator for Evaluation. 

B. The lists of questions are reviewed at the regular monthly meet- 
ing of the Evaluators with" the Administrator for Evaluation" 
reviewed for the purposes of deletion ^ addition^ changes and the 
compilation of a single list* 

C. Following the above meeting, and prior to the next monthly meet= 
ing, each Evaluator selects a few cases to review with the single 
list for appropriateness of the questions ^ and again completes 

- his or her own list» which again ^.s sent tc each of the other 
Evaluators and to the Administrator of Evaluation for review. 

D. At the second meeting the Evaluators again review the questions, 
together and complete a final list of questions with standards , 
with categorization of areas of prime importance^ using 3 if 
possible, an existing base di information :^uch as R-SOO. 

^ E. Following th^ second meeting each Evaluator pretests the flnal^ 
questions and standards by reviewing 15 cases of his or her own 
choosing. 
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F, At a third meeting, the list of questions used in the 15 case ^ 
ravlews are discussed. A question item analysis of the Case 
Review Sheet is performed for uniforra Interpretation and in- 
creased interrater reliability. 

G, The Case Review Sheet and Standards are dram up and completed by 
the group as a whole at this third meeting, 

H, The final draft, of the Case Review Sheet and Standards is= pre^ 
sented to the appropriate Central Office management level for its 
review and input, and to a representative sampling of District 
Adndnistrators for their review and input, 

I, Office of the Administrator of Evaluatfion finalizes the Case ^ 
Review Sheet and Standards. 



III. Case Review 



Meeting of Regional Case Service Evaluators to rainforce Inter^ 
ratf3r reliability, ^ 

1. At 3 regular monthly meeting of the Evaluators, or a special 
meeting called for that purpose, there is discussion of 
inf:erpretation of policy and procedure. ^ 

2 * Standards and Case Review Sheet to be used are thoroughly 
reviewed, 

3, Evaluators review a case with the developed Case Review Sheet 
and Standards. 

4, Question interpretation for Case Review Sheet la developed. 

Selection by office of Administrator of Evaluation of Cases to 
be reviewed. 



>uter is used for selection of random and stratified 
sampling of cases to be reviewed, 

2, Printout of client information is obtained and given to Case 
Service Evaluators, 

3. Assistant Director of Field Operations notifies Regional 
Administrators and District Administrators of date of review, 
and the Administrator of Evaluation provides them with a list, 
of cases to be reviewed. 

When tfase Review is completed, one copy- of the Case Review Sheet 
is given to the District Administrator ^ one copy is sent to the 
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Utilization of Case Review 

A. Conference of District Office md Regional Office Personnel 

1. Allowini at least three weeks of District Office staff to 
review evaluation findings, the Regional Administrator 
establishes a date for a conference with District Of fire 
personnel including the District Administrator, Assistant 
District Administrator and District Office Supervisors 
Regional Office personnel are the Regional Administrator 
Assistant Regional Administrator and Regional Case Service 
Evaluator. i-vj.i_e 



and 



2. Evaluation findings are reviewed and discussed— strong ai 
weak areas identified, problem case service patterns and 
areas pinpointed, need for clarification or modification of 
procedures studied, etc, 

,3, District Office gives Its evaluation of the evaluation re- 
sults, and presents the procedure or method it intends to 
use to disseminate and utilize evaluation results and to 
Implement corrective action where necessary. 

^. Procedures to be used for follow-up are discussed. 

B. Report of Case Review and Evaluation Conference by Regional 
Office ' " 

, Following the Case Review and Evaluation Conference the Case Ser- 
vice Evaluator and Regional Administrator submit a report on the 
conference to the Administrator of Evaluation, which report 
Includes: 

1. Identification of problems in and recommendations for solu- 
tions to such problems in case service areas. 



The expressed opinions of the District Office reBardluB 
Case Review. . - 



the 



An outline of the Wstrlct Office's plans for disjemlnatlon 
and utilization of i^e Evaluator 's findings to supervisors' 
and counselors, and plans for implementation of corrective 
action where deemed necessary. 



57 



C. Repov t of Administrator of Evaluation 



Following the receipt by the office of the Administrator of 
Evaluation of all completed evaluation review forms. District 
Office Sumary Evaluations* and reports of Bistrict Office Case 
Evaluation Conferences, the procedure is as follows s 

1, The material from the case reviews is compiled, reviewed and 
evaluated, 

2, The Administrator of Evaruation, after analyzing the case 
review datas drafts a report of concluslDns and reconmenda- 
tions. 

3, The draft is presented' to various Central Office management 
levels (Director, Assistant Director, Planning Section, 
Field Operations, and the Training Section) for review and 
input. . 

4, To obtain a user's perspective several District Administrat- 
ors asked for input on both content and format of the draft* 

5, The final product, written by the Administrator of Evalu- 
ation^ is distributed to the Director, Assistant Dlvector, 
Planning Section, ^ Field Operations^ Training Section^ 
Regional Administrators and District Administrators* 

D* Implementation of Changes 

1* The Administrator of Evaluation and- staff meet with Assistant 
Director of the agency aid appropriate central office staff 
to discuss implementation of recoimnendations based on the 
final report. Final decisions on acceptance or rejection of 
recommendations and Implementation of changes are made by. 
,the Assistant Director. 

1 ^ 

I 2, The staff of the Administrator of Evaluation and the Region- 
al Case Service Evaluators are available as resource people 
"-n thosr involved in the implementation of changes* 

V, i-oi.o--- • ..^ ' ■ 

'Following completion of the implementation procedures, the Regional 
Case .Service Evaluators^ as directed by the Administrator of Evalu- 
ationj spot check to determine if directed changes are^ in fact, 
being implemented* 

To date, in Pennsylvaniai we have been urillzlng our Case Review 
Process for appj^oKimately three years j have gained meaningful eKperiences 
and added another dimenstion to ^ur evaluation program* The fact that our 




experience with Case Review has been positive is due, .'eel in a laree 

" P^^''""^ and preparation, trained personnel and th^ " 

reSlea^L "7 ^'-^Sement people. For any evaluation program to have 
real meaning and va.ue, a strong coimnitment by top management is an 
?mportant"ir.'"T"""/" f """^te in Pennsylvania to have this. No less 
insures that "h "^"'""«,ff '^^^^^ «aff of case reviewers which not only 
With tL 5f 1. ""^--."^ correctly but establishes credibility 

with the field statf, another important factor in the success of any evalu- 
ation progran,. The four full-time case service reviewers were trSLd ?or 
approximately three months befpre assuming the. responsibilities oftheir 

cl^es'Sr" ^^^'^^"fl Pl-"i^i preparation that is necessary L- 
eludes the preparation of the field staff to accept the evaluation prLaL. 



• i^^^^^f Hoffer states in his book. The Ordeal of Chan ge (1961) '■Even 
in siigh. things the new is rarely without some stirring^foreboding " 

Hoivfr'tf' " '^"^ f ^^^^ tyP^ °f revlLs. 

However, If an agency develops and implements a structured and permanent 

boPn""'^? ^here may be some "stirrings of fore= 

level?'nr<or P"^f ^ ^'^^^ '"^ ^"isest an awareness type of training at all 
of tiL ti!^ impiementation of the case review process. The purpose 

or this training wouIq be. to present the "how, why and where" of the case 
^luT/f personnel are aware of the purposes and procedures there 
rht r. V '^ °- In Pennsylvania we produced a video tape of 

the Case Review Process through our Training Section. This video tape was 
shown in each of our 15 District Offices. Following the showing of the 
tape a team of BVR personnel made up of Individuals from Case Service Sec- 
tion, Evaluation Section, and Training Section were available for questions 
Distrlc^^nff • these meetings were all personn^.l in the 

very Jositive!^' experience with this training and the outcome were 

^ The Case Service Review process, like program evaluation in general, 
i« neither simpie nor easy, but in Pennsylvania we have found it well 
worth the effort. 
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Human wants and desires are unlimited but the resources to satisfy 
these wants are limited or scarce* Scarcity of resources then 
necessitates choices or preferences. The process of choice^making 
creates the notion we call "the opportunity ccst." The opporcunity 
or social costs are defined as "the value of resources that would 
have been available for other uses had special services not been 
rendered" (Conley, 1969). To make effective decisions on resource 
allocation, be it in the private sector or puhlic sector, management 
needs techniques of program analysis and evaluation. 

Benefit^cost analysis seeks to identify Investment projects or 
decisions chat will "maximize the present value of all benefits less 
that of all costs, subject to specified constraints" (Prest & Turvey, 
1965). Expressed in the language of welfare ^economics , the objective 
is to rank alternative investments in order to select those which, 
have potential for yielding Pareto optimum — an improvement that 
makes at least one person better off and nobody worse off. 

Investment decisions in the private sector are fairly .easy to 
make because the costs ^nd benefits of alternative courses of action 
and alternative resource uses can be measured with reasonable accuracy. 
However, investment decisions in the public sector, particularly in 
the area of human resources development, arel difficult to make due to 
the imperfect nature of the market in the field of human resources/ 
In the development of human resources the net benefits may be high, 
■ but they are frequently so diffused that many single individuals cannot 
capture enough of them, or have a sufficiently long->term perspective, 
to justify the additional investment even when high and quick returns 
can be captured. Then there are difficulties of measuring qualitative 
and non-economic attributes. Yet, a quantitative evaluation of benefits 
and costs of alternative programs is ^surely required for rational 
decision-making in government planning. Benafit^cost analysis, 
therefore, can only be one test of the programme worth to be balenced 
against the broader humanitarian and social considerations. 

HISTORICAL BACKGROUND OF BENEFIT^COST ANALYSIS 

The origins of the hnman capital theory He in the work of 
Sir William Putty, a 17th century English economist. He tried' to 
estimate the value of human capital by using annual per capita income 
interest and the number of Englishmen during his time; and then using' 
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his estimate to determine the capitalized value of human 'beings lost ■ • 
by war, disease or migration. ^ ^ ■ 

... \ . ■ • . , . ^ 

i„d VJ'^H '^^ ^''^^^^ ^" exceptional growth of manpower (employmint 

and crainmg) policies occurred during the decade of the 1960s Th^e 

substantial °^ ^"^^^ ^° ^^^IP cin-blt ' 
substantial unemplo>-,nent and long-run structural imbalance in the economy 

n^L ^^P^^'l throughout the decade. There was a growing 

need tq^evaluate these manpower programs to determine their impact on 
tne economy Although perfection is far from achieved and e^aJua ion 
. techniques have yaried tremendously, a large number of economic 
evaluations have^ been made in the manpower irea (Palomba, 1979) . 

■ .Evaluacions in the manpower area have at best looked at the 
economic impact upon program clients of various manpower programs ' 
' ' ' heL V °^ individuals, society and government! lone of 

^ m a .':f thfnon -asure either^hrTOTHcFlcSomi::^^^ 

impact, or the non-economic impact of manpower programs. Three recent 
major studies have reviewed hundreds of evaluations ,of manpower programs 

oL study (1975). These reviews higJ.light four major ooneluBiona; 

1. Manpower programs have been effective from an economic ^ 
viewpoint. The poverty gap has been reduced Jbut not 
eliminated for the clients of these programs; 

2. Skill training and job develbpment programs are mor- ' '. 
successful from the economic viewpoint than employability 
development or work experience pro£rams. ; Hioyaoiiity 

3. The most uconomically :ifflclent programs have been 
vocational rehabilitation (VR) , vocational education/ and iteTA 
(Manpower Development and Training Act) skill training 

■ specif lcally,on=the-job training. " ^aini^g, , 

■ ■ orSamrh'"' f °' economic gains of manpower 

houfs of ^"w^ " '° employment feffecfi (Increased 

hours of work), as opposed to a wage effect (Increased 

level or wages) . ' » ^ 

. previous , Jobs to be taken bv lesser skiii»^ • -^"^-^^^e; , their 

would understate the total Lf/"""'^ Such an effeot^ 

occurs- when thfcSen fSe SSd «««« 

. - -ne Clients are .placed in jobs at the expense of non-cMents. 
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Any increase in the client's Income will be due to more hours of work 
I (employment effect) and not due to a wage increase. The displacement 
effect would overstate the total impact of a program. 

The reviews of manpower programs show that a large part of the 
change in income is paused by an employment effect* Therefore, future 
evaluations of manpower programs must attempt to separate wage and 
employment effects, 

BENEFIT=-COST ANALYSIS IN VOCATIONAL REHABILITATION 

\ . . ^ ^ . ^ ' -'A 

Thf growing use of the planning^programming-budgeting (PPB) 'system 

in evaluating public expenditures has been accompaiiiied by a growi^ig under-- 

standing of the complexities of the methodological and conceptual problems 

facing benefit-cost analysts. Several benefit^cost studies of the Federal^ 

state rehabilitation programs as a whole or of its component parts hWe 

been undertaken-since 1965. These have been used primarily to^ win siipport 

-Jpr increased program expenditures (Noble , 1977), ^ V 

/ v^n recent yuir ^ there ihave been both internal and external motivations 
for tHe^-development of program evaluation efforts in state VR agencies^ 
Externally^ among others, there has been Increased demand for accountability 
in human services, . The 1973 Rehabilitation Act contains a requirement that 
evaluation standards be devised and Implemented to measure the ^ performance 
of the VR program in achieving its mandate. The program evaluation 
standards include *'banefit-cost and cost effectiveness" as one of the 
performance standards. Also, there has been a growing emphasis on / 
data-based program development. The goal of program development is / 
to design optimal programs which help clients achieve specific outcom^fe 

^utlli^ing available staff and resources, 

Benef it=*CQSt analysis lends Itself to predictive uses. That is/ 
it will^ tell us the! results of alternative allGcations of resources /within 
progranm and to a lesSer extent between programs (Sewell, 1965). 1 Some 
program administrators may have fears that too marrow an ecQnomic / 
interpretation of the comparative benefits and costs of programs which 
have important social and human values might undermine and distort^ both 
the objectives of programs and endanger their case for expanding Ihem* 
However, in practice, it has not been possible to use benefit-co/t ratios 
to compare VR programs except in the most general terms, Eath VR ^^ 
•program has its own set of internal and external benefits aifd cbsts and 
each has its own model built to fit its particular circumstancL , In 
general, therefore, the benefit-cost models are much more usefiGl for 
altering or otherwise improving the effectiveness' of a single/VR . ■ 

program than they are for making comparisons between programsL 

Furthermore, benefit-cost analysis imposes a very valuable discipline 
lOa^the deciaion=making pr/oceases of program administrators and policy- 
makers. It requires a cl!ear articulation of objectives (for the purposes 
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of quantification) and a precise methodology for determining exactly how 
these objectives are being met. By Identifying the amount and incidence 
bf the benefits and posts for, the clients, society and the government, 
the benefit-cost modkl will ptovlde a structure for analytic interpretation 
which will facilitate program monitoring. The mere act of gathering the 
data and etructuring the analysis for a benefit-cost model will help 
_ identify the problem areas within the VR program and facilitate changes 
, m the combination of factors employed, or in the level and structure of 

services offered to the clients CMaJumder, Greever, & Palomba. 1978). 
, .The applications of micro benef lt-co,st analysis based on continuing 

'program bfetteflts and cost data are numerous. For example,' by comparing 
, the client's past work history with his/her activities and earnings after 
VR training we can judge what kinds of clients use the program to greatest 
advantage. Do older clients do better than younger? Do the crients in 
fact use the VR training? Do they, increase their income over time as a 
result of the training? Are they less likely to be unemployed? 

All of this kind of analysis and the decisions which' flow from it must 
be undertaken with the full realization, of the practical limitations of 1 
benefit-cost analysi$, as opposed. to ultimate theoretical limitations. No 
benef j.t-cost model as yet measures precisely what it purports to measure 
because of the need . to use proxies on both the benefits and costs sides 
of the model. . ' 

Some effort Is being made by the Research and Training Center Staff 
at Baylor University to measure the qualitative benefits accruing from 
.rehabilitation using, psychometrics. . When their. study Is "completed it 
might provide some ibsight as to how some of the intangible benefits 
resulting from rehabilitation could be assessed. Of course it should 
be remembered that this' will , still not solve the problem of determining 
the range and scope of total benefits ; coniblning both market and 
non-market measures of benefits into a single global measure. But It 
is certainly a step in the right direction. 

\ ' . • ■ ' 

BENEFIT-COST ANALYSIS: WHERE ARE WE AND WHERE ARE WE GOING? 

We can summarize our thoughts by saying that benefit-cost analysis 
has come a long way in just ,lwo -decades, but as an evaluation tool 

, there IS still major development work to be done before it can ibe used 
extensively. The major problem areas that remain include; (1)' developing 
an inexpensive but reliable control mechanism so that we can accurately 
measure economic benefits-and costs: ■ (2) developing a method tb estimate' ' 
the indirect economic benefits and costs (vacuum effect and displacement 
ettect); and (3) developing a method to Incorportae non-economic 
benefits into the benefit-cost evaluation tool. It will take at \least 

,,two more decades to solve these three problem areas. ' ' ^ 
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Problem area (1) can 'be solved by using various hefore-after control 
mechanisms. and check?' g the results periodically with a live control group 
until inexpensive control mechanism is foundT which is reliable. 
Since live control groups are so expansive at best they can only be 
used periodically. Thus, finding a bafore-after control mechanism that 
can roughly predict live concrol group results at a fraccion of the 
cost is imperative. We are optimistic that this can be done. 

Problem area (2) will be more difficult. Here attempts will have 
be made to estimate wage effects and employment effects in the area of 
economic benefits so that a reasonable estimate can be made of vacuum 
effects and displacement effects. Until this problem area is resolved^' 
evaluators will have to be content to use .benefit and cost data for 
clients, only (direct economic benefits and costs). 

The third problem area will be the most difficult to handle. /it ^ 
could be quite som^ time before anyone can properly combine economic 
benefits and non-economic benefits into one global measure of benefits. 
This lis the most serious fault with benef it-cqst analysis. The/problem 
IS that while economic costs are probably the only non- trivlal/costs . of 
a social. action, program, the economic benefits arS' probably np4 the only 
non-trivial benefits. One approach in this area mky be do cihduct' an 
economic, benefit-cost evaluation, and then simply list non-eLnomic 
benefits as seen fit.. Thus, at some ifuture point in time 'Ivaluators 
might be able to make statements' about a social action prog'rams su^mst*^ 
this program cost $1 million to treat 400 clients; however.l^e-rttim^tr--— - 
that the 400 clients will earn„$600,000 more during their lifetime 
their divorce rate will be 10 percent less and their feelings toward ' 
themselves will be 8 percent higher according to some acceptable evaluation 
scale. . . . \ 

Benefit-cost analysis Is- too good not to use but not good enough 
to use ei^clusively . : - e 

\ . , ' , • ■ . ■ ■ ■ 
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PUTTING THE HORSE BEFORE ^£, 0^1.! OR, A CASE FOR VR AGENCY 
MANAGEtfflNT INtORMATION' SYSTEMS BEFORE EV.\L mTION """ 

Mississippi Vocational Rehabilitation for the Blind 



Evaluation— what It Is and is riot, what it doas and does not Why It 
is, necessary and why not. what it should and should not consist of, even 
the definition for the terin"evaluatlon"~is the subject of much debate 
and concern in the world of ^ vocational rehabilitation. Increased demands ' 
for accountability from both the Federal and state levels of government ' 
make evaluation of VR programs expediently necessary if agencies expect 
continued and/or increased funding. The state-of-the-art in VR presently 
leans toward evaluation outcomes as valid indicators of an agency's success 
or lack of success In rehabilltatlon of clients. Thus, evaluation as a 
measuring device, is being embraced by VR as never before. This new aware- 
ness of and appreciation for total program evaluation is' in Itself deslr- ' 
ablei however, the agency that attempts total program evaluation prior to 
developing, testing, and installing a formal system of managemant informa- 
tion IS In for a rude awakening..; , A logical rafutatlon of this statement,/So 
IS the fact that s^me VR. agencies have done credible Jobs rehabilitating 
clients without management information systems and' these t^encles can pro- 
duce credible statistics to verify their consistency of 26's. So then, 
why is a management information system a necessary pretequlsite for total 
program evaluation, and why does program evaluation in the absence of a 
sound management information system constitute putting the cart before thi 
, horse? 

First, let us agree that Just as a rose by any other name smells as 
sweet, evaluation, by any other definition. Is still tha appraisal of an ' 
activity to determine its significance or worth. Admittedly there are" 
agencbr staff evaluatorsj consultants from specialized firms, universities 
and research and development centers, and other experts who have much more 
flowery definitions of evaluation and who, in !all probability, have their . i 
own unique approach to evaluation. ~ While these definitions and approaches 
to evaluation are not being iichallenged, it appears that the simple defini- 
tion of evaluation herein cited and the approach to evaluation- through a 
sound management information system have equal merit. As to a management 
information system let us define, it in simple terms as well by saying that 
in. a VR agency a management iriforraatlon system Is the method of providing 
to decision makers the information they need to plan, execute, and control 
operations of the agency. For example, what information .does an agency's 
fiscal officer need to plan the agency's annual budgeti or, what informa- 
tion does an agency director need to determine the geographical distribu- 
tion of counselors in a dtate? The answers to > these? questions , and the 
method by which these answers are provided, constitute a management ih-\ 
■formation system. Ip all likelihood, VR administrators have all the' in- 
formation they need, but this information is probably riot, supplied in a 
form which is usable or aystematic enough , for it to be valid. Thus, 
managers at any level in VR need some concept of what information is 

: ' ■ ■ ' ' ■ I . '■ ; , ^ ' ■ 
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„ nf^m^ in whan form It is naeded, criteria by wWch-'to av^luate the in- 
formation, and some concept of , timallnass. .Summarily, a management In- ' 
formation system can, be as simple or as complex as the agency manager's 
demand based on their declslohmaking needs. What of the agency with its 
proud history of consistent 26s and no management' Information system' The 
agency simply does not exist. While there may have been no formal (written^ 
plan for management information, some management information system has 
been in operation because decisions have been made to produce, the record 
of 26s. The system mny have been the Intuition of top managempntj the 
collective intuition an oligarchy, or benevolent dictatorship— but a ' 
System existed! . ' 

. . ■ I 

Now that we are armed with simplistic, though sound, definitions of 
the terms evaluation" uad "management information system" we can regress^ 
to answer the original question: how does attempting total program evalu- 
ation prior to Installing a formal management information system constitute 
putting the cart before the horse? 
■ ' r ■ ■ ^ 

Th& existence of a formal management information system (formal mean- 
ling written) denotes the existence of several important conditions 1 which, 
in turn, permit objective program evaluation. Basically,' these conditions 
are: (1) a well defined organizational structure} (2) a clear understand- 
ing of the legal responsibilities of the agencyj (3) a method for system- . 
atic planning involving input from all agency strata; and (4) a well 
defined mission st/atement supported by realistic, flexible goals and measur- 
able objectives. Before examining the importance of each of these con- 
ditions, it should be noted that there are various types of management in- 
formation systems and the one an agency o^ts to use depends on factors 
peculiar to that agency. . Regardless of the type of management information ' 
system developed, one factor is critical in all of them: willingness to 
accept change. AgencV management which is oligarchic or benevolently 
dictatorial will not/find ,tt easy, to cite the old cliche, to give up the 
bird in the hand for the possible two qr three which management information 
iindicates are possibly in the bushes. Yet, the purpose of a management 
Information system is to provide, decisionmakers with Just such inforiAatlon, 
and the purpose of evaluation is to determine the significance or worth of 
the proposed activity which might yield two birds as well as the activity 
which has proven to yield one bird. Thus evaluation and management in- 
formafiion, though strange bedfellows, complement each other. 

As one of the conditions resulting from a management information ' 
system, what does organization structure have to dp with evaluation? Firat, 
and perhaps foremost, organization structure in the VR agency delineates the 
allocatioris of tasks necessary for the rehabilitation of clients. In VR^ 
the rehabilitation of the client is not the 'sole purview of the counselor j 
rather, each agency staff person contributes to the client's rehabilitation 
.through the performance of his assigned tasks. This interdependeney element 
makes it necessary that each staff person have an assigned place In the 
agentty- hierarchy and a written job description which clearly outlines his 
responsibilities, to whom he is reaponslble, and, how the successful 
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tSr^i n of -his assigned tasks Impacts the tasks of his unit ^and of 
the overall agency Organiaatlon'al structure further insures that 

. management has a firm understanding of tftaff duties, that management dls- 
sS"tu«"''°-n inanagement,. through organisational 

a^f^^ ^f * jan allocate tasks resulting from the levy of new mandates or ' 
Eatl^if f ^ 'V*"" 1^ Looking at organic ■ 

. servef L futth^^^^^ it is a functional too! which 

ehln h.^^ ' - °^ rehabilitating clients, the relation^ 

■ ,^^^"''--",°^8anization structure and eval^^^ 

standable. Thus, whether an individual staff member is being evaluated to 

, determme^whether or not he has successfully contributed to the agency 's 

purpose through compfLetion of his assigned tasis; or, whether the organi- ^ 

serS"^J """""^^ being evaluated to determine how well it 

structur^ f * ' ^'^i' ' «^ organisation : 

structure .that Is supportive of a system of man, jement Information has a 

high probability for being successfully evaluated. j 

changes in policy affecting local VR operations are handed 
2 wefr^f^ i SovernmentUnd in some instances, the state government 

is ^e ;f the loL^^ T""''"* °' "8"""^ i« l^S^^y "sponsible for 

is one of the bonuses of a management Information system. ' Decisions as to 
how an agency handles Federal and state mandates should be reflected in the 
• agency manuals or^handbooka. which means that once again a n^anagament in^ ' 
formatxon system is needed to systematically take caxe of such situations. 

^i^^^tanding of what an agency Is legally responsible for ■ 
facilitate evaluation? This, question can satisfactorily be answered by 
citing that^ the .Mississippi mu, in the process of developing a Basic — 
Management Paradign for Mississippi Vocational Rehabilltatidn for the 
Blind, developed a Compliance Activities Matrix consisting of some twenty 
(20) activities. Federal, antf 'state, for which thd agency is responsil^le 

"v!"* ^ y^^^- Comprising the agency's legal respon- 

sibilities, these twenty (2) compliance activities run the gamut from 
continuing studies to required reviews. In a management information system 
such a matrix provides management wlthi (1) a listing of all compliance i ' 
activities! (2) required completion dates whldi provide management with 
basis for determining initiation dates; (3) a basis for allocating 
staff and time necessary for conducting the compliance activity! and 
(4) periodic review and reappraisal of agency approaches to conducting 
tasks necessary to comply with mandates. . e 

Evaluation of an agency must necessarily address whether or not the 
agency is In compliance with Federal and state mandates—legal responsi- 
bilities. A management information systems makes the process of evaluat- 
ing this facet of. an agency's operation a less complicated task. It should 
be noted here that a clear understanding of an agency's legal responsibi- 
lities does not 'exclude those responsibilities which the agency sets for 
Itself. In fact, when Wlewing the Mississippi Mu's Compliance Activi- 
ties Matrix, a number of' other areas of agency Interest were discovered. 
The VR agency genuinely concerned with its mission of rehabilitating the 
client will naturally move to independently take on the "discovered" , 
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!m^h °^ i^^^'^^st. and evaluation will also. reflact those agency projec b, 
which are che outgrowth of legal responsibilities. 

Systematic planning^ involving input from all agency strata is anot ,er 
essential feature of a management information system which impacts the 
-evaluation process. To assure that each staff person in an agency recog- 
nizes his 30b assingment as supportive of his component's tasks and 
necessary to the success of the agency in rehabilitating clients, a manage- 
ment information system should encompass all agency levels In plannlnR 

SnL^^f^'^^^J^-./^f^-''^ ^ ^^^^ ^ trl-fold process, ihdividual 
planning f^or individual activities: small group planning for componek 

^^IIJ -^^f'^"?^' f'"^"""atlve planning for the overall agency. Admlnia- 
tratlve planning fop the total agency should utilize the input of the small 
group . (component) and the indivudal. It was noted earlier In this article 

"P-^- ^" ^Sency contributes, directly or Indirectly, to 
the rehabilicatlon of the client; thus, each staff person has a' vested 
interest In the agency s overall success. Planning from the Individual 

< tTt "P to and through the administrative level should reflect activities 

which will lead /to the ultimate end 'desired by tha agency 



It IS iMossible to engage in planning without a clear understanding 
of an agency/Sj.^gal ■responsibilities and .without an organizational structure 
which permibff input of ideas from, one level to the next. When agency 
planning is evali^ted, the input of , ideas should be feaslly traceable from 
top to bottom, or bottom-toptop. Additionally, evaluation of agency planning 
witHin the confines of management information can move past addressing 
quantitative ^tems such as how many activities were planned to 'addressing 
qualltatiye aspects such as how well were activities planned, to what degree 
were the plans initiated and completed, -and what impact did the planning 
and conduct of the activities have on the agency's desired end._ 

It wbuld seem logical that before an agency could begin total program 
evaluation, I it should have in writing, and as part of its management in- 
tormation system, a mission statement, realistic yet flexible goals,, and 

. measurable objectives. When queried as to the mission of VR, the stock 
and truthful response is "to rehabilitate people.'^ It is therefore in the 
best Interest of VR agencies to put in writing their plan for ..rehablll- ' 
tatlng people. , If the plan is hot in writing, how can it be evaluated? 
If an agency has a management information system, it Is virtually impos- 
sible not to have a mission statement, goals and objectives. One could, of 

•course, ^fall back on .counting nhe 26s, but we Have already established 

that this alone does not equal total program evaluation. ... 

Establishing a mission statement, gOsils and objectives as patt of an 
agency's mai^agement Information system is not a difficult matter. In fact, 
most VR agencies probably, have a mission statement, goals and objectlveb 
stated in some form in the various documents they produce. The mission 
statement, goals and objectives of Mississippi Vocational Rehabliitation ' 
{for the Blind were culled from the agency's annual reports, brochures, 
state plan, etc. As^ Illustration, consider ths following ' diagram which is 
■ ./ I , I / , • ■ 
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taken from the Basic Manggement^ Paradigm being developed by the Mi 
:iMEU for Missiasippi Vocational Reh^illtatlon for the Blind i 
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HAlNTAINiNG RELATlqNginPS FftOM HISSION STATEM ENT TO QBJRCTIVE SPECIFICS ' 

A slmpU mt). V mmd by the E/M Unit to maintain relationehlp from 
miSEJOn gtatement,-fioa1 -objective and objective spgeJfics is ilJuatrated hera 
The mt thodig simply to move from the broadest point the moit specific point J 



The Miioidn Statement 
i B the moGt general & 
its accompl ishifltnt is 
expected to cover a 
lon^ period of time. 



WiTHiN THE PARAMETERS 
OF THE REHABILITATION 
ACT OF 1973, ENABLING 
STATE LEniSLATl ON, AND 
ANNUAL LEVEL OF FUNDING 
MliSlSiiPPI VOCATIONAL 
'REHABILITATION FOR THE 



BUND WILL pROVlOE SUCH 
REJf ABI LI TATTOwlsEn VI CES 



A3 HAV BE NECESSARY TO 
CLIENTS TO AFFOBD THEM 
THE GREATEST POSSIBLE 



OI*J>URTUN ITV WjBUQQEm 

THE} PURSUIT, ATTAIN- 
MENT p AND RETENTION OF 
I A VOCATIONAL OIUECTIVE 



the Goal Stntemont 

It is sufiportive 
of the miajilon 
in that its 
accomnlirihment 
impacts misaion 
accomplishment 



The Objective 
is the if aat 
general and its 
accompli shment 
impacts both 
goal and miagioh 
accomplishment* 




Objective 
Specif ice 
provide 
measurables 
needed for 
evaluation 
activi ties , 
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agency „aatlvlty in ^^"^^i """f-"^ 

because It la a written manaBement Dl«n - % ' rahabllltatlon; It permits, 
and it gives the evaluato^ aometM.^ f * ^e^i*^ and' revision as necessary; 
praising agency activities. ' ° ""^^^ ^^'^^ process of, ap- ' 



be mJLiL Pu' ^nf^^^^ can only measu« that whicb axl.ts to 

coming state of the art ° ^^at quaxitatlve evaluation- is the 

qualitative. «Lsu'abI^f satS^^ information systems again deal with ' 

s.rr-„n^.— ^^^^^^^ 
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BOOK REVIEW 



1 



I A ^DBOOK FOR FOLLOW-UP STUDIES THE HIMM SERVICES 

BY KENNETH W. REAGLES, SYMCUSE UNIVERSITY, N.Y. : 
THE ICD REHABILITATION AND RESEARCH CENTER 

NAN E* BRENZEL, ED,D, " ^ 
WEST VIRGINIA RESEARCH & Tr4iNING CENTER 



ThB ait tempt to discuss the areas cousldeired by Kenneth W. Reagles in ^ 
A Handbo ok for Follow-Up Studies in the Human Services is welcomed by those 
designated the responsibility of evaluating human services, keagles 
states that the book is *'not intended as a guide for conducting follow-up 
studies, it is not merely a "cookbook;*' there is liberal discussion of the 
reasons and rationale for various survey research methods," In accordance 
with the author -s stated purpose, Dr, Reagles relays sound technical in- 
struction evidenced by the wisdom gleane^ from years of program evaluation 
experience. However, the Handbook does not stop at the technicalities of 
conceptualization, formulationj design and eKecution of follow-up studies* 
In addition to d^italled precise Information the Handbook offers "pointers." 
These "pointers^" again evidence of eKperience (yes. Ken Reagles had a day 
when he began learning about program fjyaluatidn too!) are provided to the 
reader in a respectful manner. For the novice program evaluator the 

pointers" elicit the "aha" response. For the more advanced program evalu- 
ator the "polntera" servje to substantiate tlie ierlousness and depth of 
involvement required by program evaluation, Reagles* style of presenta- 
tion demonstrates that follow-up studies not only require technical expert- 
Isti, but also a willingness to draw from one's personal reservoir of 
creativity, Reagles demonstrates this continually throughout the Handbook . 
An appreciable addition to a program evaluation type book is the author's 
use of humor. Reagles' use of humor is exquisite ! It serves to dispel the 
myths that program Wvalufttlan is conducted by egg-heads who memorise \ . - 
formulas J procedures and statistics. . 

The Handbook is designed primarily f or^ thoi , with little or basic 
understanding of program evaluation. The Handbook can serve well as an 
introductory teaching tool for new program evaluators or as a reference and 
task checklist for "old timers" in the program evaluation buslneBS, 

' . Although major emphasis is placed on follow-up studies in the human 
service field, Reagles' style is exceptionally noteworthy for versatility. 
It affords applicabili^ty of materlar to a variety of disciplines desiring 
to conduct follpw-up evaluation, 

A technical presentatiori of mate'^lal combined with the on-.target . 

propositJon of content questions and the personal flare of this writer 

provides a conversational tone which enables readers to feel that 
Dr. Reagles is talking directly to them. 
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Although, the primary purpose of this Handbook "is to enhance the' 
capacity of such agencies (human services) to conduct follow-up studies " 

cl^rJti °' '^8* °' evaluation materials off 

clarity, understanding and enjoyment of a once dreaded subject area For 
Chis purpose %ach chapter In this book deserves separate cofflnenl! ■ 

Chapter^one Includes an introduction which addresses topics such as 
pressures and purposes of program evaluation, purpose of fonow'up studies 
and^.nformational needs, studies. Reagles speaks to the Issues of reJuJred 

^^d.rr 1°''" "^"^"^ Pf trained personnel 

needed to conduct evaluations. This chapter set^ the tone for tje serious 
recognition which program evaluation in human services deserves. 

Chapter two discusses the steps in planning a follow-up study. The 
(author provides an adapted list of general considerations for pll^'ing 
The list begins With ways to decide on the purpose of the study and some 
22 steps later concludes with quebtions on utilization of findings and 
recommendations for further study. A flow chart (p. 24) j illustrates pro= 
gressive steps in the formation of general questions about a program to 
specific Items on a follow=up questionnaire. It is in this chapter that 
. the autnor makeswhat reads as a simple statement but perhaps contains the 
essence of program evaluation integrity, "no evaluation is bet^r than a 
bad evaluation." (p. 25) niei LUdn a 

Chapter thr^ee provides an inforrnatlve discussion on the/ypes of sur- 
vey studies. Reagles, offers desprn^tlons of each type surve^ with listings 
of advantage, and disadvantages fcr use. To complete what^li already befn 
a comprehensive presentation of the mailed questionnaire, ^rsonal inter- 
view, telephone Interview and other data collection methods (observational 
approach, case file review, combinations) the kuthor provides , the reader 
with a magnificent modal for evaluating the utility of a specific surv4 ■ 
method (p. 60-61). The model lists the survey method and provides fcon- 

n'f M^h°"' r^^^'""^" ^ith a + (advantage) , - disadvaJtage and 

0 (neither advantage nor disadvantage) rating. In. this chapter the author 
cautions the potential user of surveys that "regardless of the approach" or 
method of data collection, each is only an approxlma'tlon of reality." . 

Chapter lour entitled "Constructing an Instrument" provides detailed 
planning formats, considerations for questionnaire construction, principles 
of Item construction and layout instructions. Chapter four is clear and 
concisr The clarity is partially contributed to by the numerous examples 
which low each content area. Advantages and disadvantages for the use 
of a particular method are also listed. To round out a laudable explana- 
tion of instrument construction Reagles adds "a note on reliability and 
validity (p. 97). While this section in. no way serves as a full presenta- 
tion bf these concepts, it does discuss what program evaluation would agree 
are the most critical concerns of reliability and validity. 

A, u °^^P^®^ addresses the subjects of pretesting arid sample sslectlon. 

Although a distinction is made between pretastlng and pilot studids by some 
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reeearchers (as noted in j^he text 99), the author chooses to treat pre- 
testing afnd pilot studies synonymously. Chapter fiye begins with the 
presentation of the purpose. of pilot or' pretests , ^jReagles raises import^ 
ant questions which require attention by those conducting follow-up 
studies. Emphasis is heavily stressed on the importance o£ pretesting or 
pilot testing* Again the author combines' an exacting explanation and 
.personal component to these concepts, as exhibited in his statement i 

"The feeling of standlngJb.ffaEa^Undreds or thousands of 
questionnaire^ to be mailed and to know that you are 
responsible is a humbling one; you will feel reassured at that 
moment If a pilot test hag been conducted," (p, 104) 

The next section of chapter five is devoted to the stages of the pre- 
test* Statistical sampling is the topic for the remainder of chapter five 
The focus is on the conceptSi the different types sampling and their re- 
lationship to rehabilitation and "other human services. For such a tradi- 
tionally misunderstood and confused topic areas sampling is eloquently, 
eKplicitly and logically presented to the reader, A short introduction of 
basic terminology is presented^ followed by an in-depth discussion of 
simple random sampling, sampling variations, stratification, clustering 
and sample size, Su^portin^ studies and relevant literature are cited 
throughout the chapter which allows pursuit of additional inforination 
should the reader desire a more in-depth study of a particular iubj act* 

Chapter slx/4s concerned with the variety of techniques that cf»-"^^^ 
used prior to conducting the actual survey. This chapter illustrates the 
critical need to assert common sense in conjunction with any^^-technicpl 
endeavor in research* The author's year^ of practical pxperlence / are. 
perhaps most evident in this chapter, Reagles substations chapter sIk 
into a discussion of i iden t:\fylng the^ pwp.ulation and sampler-advance 
notice of the survey, the use of incentives, thfe accompanying letter and ' 
mailing procedures, appearance of the questionnaire, sequence of idalllng 
operation s follow-'Up efforts and special accommodations for specific dis^ 
ability groups. Throughout chapter six the reader is given point by point 
guidelines and practical suggestions which would definitely aid in the 
tasks of data collection* Substantive research supports many of the 
author's directives* 

Chapter seven provides the nuts and bolts of personal and telephone 
interviews* The last fifteen pages of the chapter Include an example of a 
survey. Although somewhat lengthy, the sample survey does Illustrate many 
of the points previously discussed in the pr&iceding chapters. In Reagles ' 
explanation of personal interviews, the reader is constantly reminded that 
personal interviews do mean personal i;. that is per Qon^to-tperson involvement* 
The author addresses topics varying from methodology of administration to 
suggestions for dv^ss and apparel when doing an interview* In a few short 
pages Reagles has packed an enorjnous amount of infdrmatlon* However, the 
InforTOation is neither overwhelming nor confusing. It safely falls short 
of a potential program evaluation saturation points 

\ \ . ' 
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• Chapter seven s niCty-gritty approach may be overwhelming. To some 

however, it serves nicely to point out the meticulous detail involved in' 
.the assurance of successful personal interviews. 

Chapter eight, the final chapter of the Handbook pi-wides brief dis- 
cussions on data collection, storage, analysis and utilization. This 
chapter gives the reader an overview on subject areas and seems to combine 
the necessary information to stimulate the reader to pursue additional ir ! 
depth literature relevant to his/her specific needs. Chapter eight's ' 
xn ^ «npears to be only to familiarize the readerwith concepts such as 
• , - um-records, data=processing terminology, data coding format and 
AHi;. f ^ section on data analyses and cleverlv places an 

tJr;.1. t and Tiedeman (1954) (p. 211) to help organize 

TM, ^^M - °' selecting a statistical procedure to analyze La: 

Plnallv ^h^^ -ving feature to those not versed in statistical methodsJ 
Finally the author presents the reader with a final note and one that is 
worthy of quote: > 

"Fdllow-up studies represent an imposition upon those from whom 
data IS collected- as such' they must be undertaken with the 
utmost seriousness and consideration for the riglits of the 
respondents." • 



Yes, that's what 'A Handbook for Foll ow-Up Studies in the Human Ser- 
vices IS all about I seriousness and consideration with a little humor to 
keep you on your toes. , . 

, , A Handbook for Follow-Up Studies In rh^ Hu man Services is 222 pages 
(softcover) in double-spaced typed format. Thj: Handbook is liberally 
Illustrated with flow charts and examples .and is available from ICfi Rehabi- 
litation and Research Center, New»York, Haw York 10010. 
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